Doramagic Project Pack · Human Manual

airweave

Open-source context retrieval layer for AI agents

Repository Overview & System Architecture

Related topics: Source Connectors, Entities & Sync Pipeline, Search, Embeddings, Reranking & Retrieval Layer, Authentication Providers, Billing, MCP Server, Connect Widget & Deployment

Section Related Pages

Continue reading this section for the full explanation and source context.

Related topics: Source Connectors, Entities & Sync Pipeline, Search, Embeddings, Reranking & Retrieval Layer, Authentication Providers, Billing, MCP Server, Connect Widget & Deployment

Repository Overview & System Architecture

Purpose and Scope

Airweave is an open-source data pipeline platform that ingests content from external SaaS sources, normalizes it into searchable entities, and writes them into a vector-capable destination for downstream AI agents. The system is built to expose those ingested records through the Model Context Protocol (MCP), letting agents query collections in a uniform way regardless of where the underlying data originated.

The product surface spans two halves: a Python/FastAPI backend that owns the sync engine, authentication, persistence, and connector orchestration, and a frontend workspace for managing collections, sources, and destinations. The backend lives under backend/airweave and is launched via backend/airweave/main.py, which wires the FastAPI application, mounts routers, configures middleware, and registers startup hooks for background services. Source: backend/airweave/main.py:1-80

At the application level, the product concepts a developer needs to recognize are:

  • Source — a connected third-party account (e.g., OneDrive, SharePoint, Google Drive, Notion, MongoDB, custom code).
  • Collection — a logical grouping of sources that share a destination and access policy.
  • Destination — the storage backend where entities land (Vespa is the primary vector destination).
  • Sync — a scheduled or on-demand run that pulls entities from one or more sources and pushes them to the destination.
  • Entity — the normalized schema produced by a connector before it is embedded and indexed.

The boundaries of the repository are intentionally wide on the connector side and narrow on the agent side: Airweave does not embed AI itself — it prepares data for agents and exposes it through MCP.

High-Level Architecture

The backend is decomposed into four cooperating layers, each with a clear responsibility and a narrow interface to its neighbors.

LayerModule pathResponsibility
APIbackend/airweave/api/HTTP/MCP surface, request validation, auth
Platform / corebackend/airweave/platform/Connectors, entities, sync orchestration, destinations
Core infrastructurebackend/airweave/core/Settings, dependency container, persistence, queues
Models & schemasbackend/airweave/models/, schemas/ORM and Pydantic shapes shared across layers

The architecture document backend/airweave/architecture-refactor.md codifies the layering rules and the way the platform depends on core but never the reverse. The Container in backend/airweave/core/container/container.py is the composition root: it builds singletons for the database engine, Redis client, temporal client, OAuth providers, and platform registries, then hands them out via FastAPI dependencies. Source: backend/airweave/core/container/container.py:1-120

Settings are loaded once from environment variables and .env files through Pydantic settings in backend/airweave/core/config/settings.py, with explicit sections for database, Redis, Temporal, OAuth, embeddings, and feature flags. Source: backend/airweave/core/config/settings.py:1-100

Sync Pipeline

The sync pipeline is the operational heart of Airweave. Each step is an isolated unit with a single output contract so it can be retried, replayed, and inspected independently.

  1. Source enumeration — the connector's generate_entities yields entity records, handling pagination, auth refresh, and rate limits. Source: backend/airweave/platform/sync.py:1-150
  2. Transformation — entities pass through user-configurable mapping and chunking stages that produce embeddable chunks.
  3. Embedding — chunks are vectorized through a configurable embedding backend (OpenAI, Mistral, or self-hosted) — see PR #1803 that made the Mistral base URL configurable. Source: backend/airweave/core/config/settings.py:1-100
  4. Destination write — chunks and vectors are pushed to the destination; the Vespa destination implementation handles indexing, deduplication, and ACL enforcement. Source: backend/airweave/platform/destinations/vespa.py:1-120
flowchart LR
    A[Source Connector] --> B[Entity Yield]
    B --> C[Transform & Chunk]
    C --> D[Embedding Service]
    D --> E[Destination - Vespa]
    E --> F[MCP / Agents]
    F --> G[Search & Retrieval]

Syncs are durable: progress, cursor state, and per-record errors are persisted so a failed sync can be resumed without re-fetching from the source. ArfReplaySource is provided for replaying captured payloads in tests, accepting pipeline kwargs to mirror production stages. Source: backend/airweave/platform/sync.py:1-150

Connectors, Entities, and Destinations

Every connector implements a common interface declared on Entity, defined in backend/airweave/platform/entities/base.py, which guarantees a stable entity_id, webpage_url, breadcrumbs, and metadata envelope on every record. This contract is what lets downstream stages and the MCP surface treat OneDrive, SharePoint, Teams, Notion, Appwrite, and custom sources interchangeably. Source: backend/airweave/platform/entities/base.py:1-80

Recent platform work tightened behavior around edge cases that surface across many connectors:

  • Microsoft Graph sources (OneDrive, SharePoint, Teams) honor Purview sensitivity labels to skip restricted files and sites (#1790), and team-scope sharing links are attached per-file rather than as a blanket grant (#1774, #1775) — both addressing security-sensitive behavior flagged in #1807 where prefetching continued after account lockout.
  • Teams messages must be sanitized before ingestion: the escape character 0x1B is rejected by Vespa, so connectors strip control codes before yielding entities (#1269).
  • Custom connectors are a long-standing ask (#1268); the platform's BaseConnector shape is designed to be subclassed from outside the package, but no first-class registration API is shipped yet.

Operational Concerns

Airweave is designed for self-hosted and cloud deployments with the same code path. The container performs startup checks for the database, Redis, and Temporal endpoints before accepting traffic, and the settings layer exposes a single disabled flag surface for optional subsystems like the tabular collection browse view (#1780). For high-availability Redis, Sentinel topologies are recognized (#1796), and workflow failures caused by user-managed credential errors are explicitly excluded from run-level failure counts (#1787), so misconfigured OAuth on a customer side does not poison the operator dashboard.

OAuth responses are normalized in the SDK: regardless of whether a provider returns expires_in (OAuth 2 standard) or expires_at (#1024), the platform persists a unified absolute expiry on the connection record. Configurable API keys per provider (#41) are surfaced through the same settings module, allowing operators to swap OpenAI, Groq, Ollama, or Mistral without code changes.

Source: https://github.com/airweave-ai/airweave / Human Manual

Source Connectors, Entities & Sync Pipeline

Related topics: Repository Overview & System Architecture, Search, Embeddings, Reranking & Retrieval Layer

Section Related Pages

Continue reading this section for the full explanation and source context.

Related topics: Repository Overview & System Architecture, Search, Embeddings, Reranking & Retrieval Layer

Source Connectors, Entities & Sync Pipeline

The Airweave platform is built around three collaborating subsystems: Source Connectors (which pull raw data from external SaaS APIs, databases, and file systems), Entities (the normalized in-memory representation of that data), and the Sync Pipeline (the orchestrator that streams entities from a source through transformation stages into a destination such as Vespa). A sync run is a single Temporal workflow invocation that drives a connector, consumes its entity stream, and writes documents to the configured destination.

Source Connector Framework

Every connector inherits from a common base class that enforces a uniform async-stream contract, so a new integration only needs to implement data fetching logic rather than the surrounding plumbing. Source.generate_entities() is the central entry point; it yields BaseEntity instances (or AsyncIterator[BaseEntity]) so the rest of the pipeline can stay backend-agnostic. Source: backend/airweave/platform/sources/_base.py.

Connectors manage their own authentication state. The base class exposes helper methods for OAuth refresh, rate-limit back-off, and cursor persistence; concrete implementations like SharePointOnlineSource build on these helpers to deal with Microsoft Graph pagination, sensitivity-label filtering (added in v0.9.70), and per-file sharing-link scoping (v0.9.67). Source: backend/airweave/platform/sources/sharepoint_online.py. As reported in issue #1807, connectors must also handle NEEDS_SOURCE state correctly when an upstream account is revoked or locked out — the connector should stop prefetching rather than continue to call the API.

Entity Model

Entities are Pydantic-style records that carry both user-visible fields (e.g. message body, file metadata) and framework-level metadata (timestamps, source identifiers, web IDs). The base class BaseEntity defines the validation hooks, serialization rules, and embedding-friendly field typing that every concrete entity inherits. Source: backend/airweave/platform/entities/_base.py.

A concrete example is TeamsMessageEntity, which models a Microsoft Teams chat message. Because the destination layer expects strictly UTF-8 text, any non-printable control character (such as 0x1B ESC) must be sanitized before the entity reaches Vespa — issue #1269 documents a sync failure caused by an unfiltered ESC byte in a Teams message body. Source: backend/airweave/platform/entities/teams_message.py.

Sync Pipeline Orchestrator

The orchestrator is the conductor of a sync run. It receives a Sync record, resolves the configured source and destination pair, opens a Temporal workflow (RunSourceConnectionWorkflow), and threads entities from the source stream through transformation, deduplication, embedding, and write phases. Source: backend/airweave/platform/destinations/vespa/destination.py and backend/airweave/domains/sync_pipeline/orchestrator.py.

The Temporal workflow is responsible for the long-running mechanics: heartbeat reporting, retry policy, graceful cancellation, and final status emission back to the syncs domain. Source: backend/airweave/domains/temporal/workflows/run_source_connection.py.

flowchart LR
    A[Source Connector<br/>generate_entities] --> B[Entity Stream<br/>BaseEntity]
    B --> C[Transformer / Sanitizer]
    C --> D[Embedding Stage]
    D --> E[Destination<br/>e.g. Vespa]
    E --> F[Sync Record Update]
    F --> G[Temporal Workflow<br/>RunSourceConnectionWorkflow]

The SyncService layer sits above the orchestrator and exposes the HTTP API used by the front end. It validates user input, persists the sync configuration, and dispatches a workflow per run; it also normalizes sync outcomes so that customer-side credential errors are not miscounted as workflow failures (fix shipped in v0.9.69). Source: backend/airweave/domains/syncs/service.py.

Lifecycle, Resilience, and Community-Driven Fixes

A sync moves through several states — pending, running, completed, failed, or NEEDS_SOURCE — that are persisted alongside the workflow execution. Recent releases have tightened this lifecycle:

  • v0.9.67 patched SharePoint Online to scope sharing-link viewers per file and stop organization-scoped links from bypassing access control. Source: release notes.
  • v0.9.69 stopped treating customer credential errors as workflow failures, so dashboards no longer misreport transient auth issues. Source: release notes.
  • v0.9.70 added Purview sensitivity-label filtering for SharePoint and OneDrive, allowing organizations to exclude confidential files from indexing. Source: release notes.
  • v0.9.71 accepted sync-pipeline keyword arguments in ArfReplaySource.generate_entities, improving replay determinism. Source: release notes.

Community requests such as #41 (configurable AI provider keys), #943 (MongoDB connector), #1024 (OAuth expires_at/expires_in consistency), and ongoing custom-connector demand (#1268) all shape how this pipeline is expected to evolve toward pluggability and per-tenant credentials. When authoring a new connector, developers should follow the existing base class contract, sanitize text payloads before they reach the destination, and ensure that revoked-access conditions surface as NEEDS_SOURCE rather than silent retries.

Source: https://github.com/airweave-ai/airweave / Human Manual

Authentication Providers, Billing, MCP Server, Connect Widget & Deployment

Related topics: Repository Overview & System Architecture, Source Connectors, Entities & Sync Pipeline

Section Related Pages

Continue reading this section for the full explanation and source context.

Related topics: Repository Overview & System Architecture, Source Connectors, Entities & Sync Pipeline

Authentication Providers, Billing, MCP Server, Connect Widget & Deployment

1. Purpose and Scope

This page covers the cross-cutting surface of Airweave that sits between data sources and end-user applications. It documents the four cooperating subsystems:

  • Authentication Providers – pluggable backend logic that brokers OAuth handshakes and stores per-source credentials.
  • Billing – subscription state, plan limits, and the usage ledger that meters sync operations.
  • MCP Server – the Model Context Protocol (MCP) entrypoint that lets external agents (e.g., Claude, Antigravity) invoke Airweave tools.
  • Connect Widget & Deployment – the embeddable frontend component that drives the user-facing OAuth flow, plus the containerized topology that ships it.

Together they form the path a user follows: open the Connect Widget, complete an OAuth flow managed by an Auth Provider, store the resulting Source Connection, meter the resulting activity in the Billing/Usage domain, and expose the searchable corpus to an agent through the MCP Server.

2. Authentication Providers and OAuth Flow

Airweave treats authentication as a domain-first concern, isolating OAuth protocol code from connector logic.

This separation addresses the community pain point tracked in issue #1024, where SDK users hit inconsistencies between expires_at and expires_in; the normalization happens centrally in oauth2_service.py rather than at every connector.

3. Billing and Usage Ledger

The billing domain is read-mostly at request time and write-mostly at sync completion.

4. MCP Server and Connect Widget

The two halves of the user surface are deliberately thin clients over the HTTP API.

  • mcp/src/server.ts boots an MCP server using the @modelcontextprotocol/sdk, registers tool handlers, and proxies authenticated requests to the Airweave backend using a per-tenant API key. Source: mcp/src/server.ts:1-95.
  • mcp/src/tools.ts enumerates the tools exposed to MCP clients (search, list_collections, trigger_sync, get_source_status). Each tool maps 1:1 to a backend REST endpoint. Source: mcp/src/tools.ts:1-80.
  • frontend/src/components/ConnectWidget/index.tsx is the embeddable React component that a host application mounts inside its own DOM tree. It opens a popup, walks the user through the /oauth/{provider}/authorize → /callback round trip, and finally POSTs to /source-connections with the returned code. Source: frontend/src/components/ConnectWidget/index.tsx:1-130.
  • The widget exposes a onSuccess(connection) callback so host apps can navigate to their next step without re-fetching state.

5. Deployment Topology

Airweave ships as a multi-service stack orchestrated by Docker Compose, with optional Kubernetes manifests for production.

  • docker/docker-compose.yml defines services for the FastAPI backend, the Postgres database, Redis (with Sentinel support added in v0.9.72), Vespa for vector search, Temporal for workflows, and the MCP server. Source: docker/docker-compose.yml:1-90.
  • Environment configuration flows through .env files consumed by Pydantic settings; secrets (OAuth client secrets, Stripe keys, Mistral base URL added in v0.9.73) are never baked into images.
  • The MCP container is stateless; it pulls per-tenant keys at startup from AIRWEAVE_API_URL and AIRWEAVE_API_KEY, making horizontal scaling safe.

6. End-to-End Flow

sequenceDiagram
    participant U as User
    participant CW as Connect Widget
    participant API as Airweave API
    participant AP as Auth Provider
    participant LED as Usage Ledger
    participant MCP as MCP Server
    U->>CW: Click "Connect Asana"
    CW->>API: GET /oauth/asana/authorize
    API->>AP: get_auth_url(state, PKCE)
    AP-->>U: Redirect to Asana login
    U->>API: /oauth/asana/callback?code=...
    API->>AP: exchange_code_for_token(code)
    API->>LED: record source_connection.create
    MCP->>API: tools/call search({q})
    API-->>MCP: results + quota snapshot

This diagram ties the four subsystems together: the widget drives OAuth, the auth provider persists credentials, the ledger meters activity for billing, and the MCP server exposes the resulting corpus to agents.

Source: https://github.com/airweave-ai/airweave / Human Manual

Doramagic Pitfall Log

Source-linked risks stay visible on the manual page so the preview does not read like a recommendation.

high Security or permission risk requires verification

May increase setup, validation, or first-run risk for the user.

medium Configuration risk requires verification

May increase setup, validation, or first-run risk for the user.

medium Configuration risk requires verification

Developers may misconfigure credentials, environment, or host setup: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE

medium Configuration risk requires verification

Developers may misconfigure credentials, environment, or host setup: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail

Doramagic Pitfall Log

Found 26 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Security or permission risk - Security or permission risk requires verification.

1. Security or permission risk: Security or permission risk requires verification

  • Severity: high
  • Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: community_evidence:github | https://github.com/airweave-ai/airweave/issues/1801

2. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: capability.host_targets | https://github.com/airweave-ai/airweave

3. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE
  • User impact: Developers may misconfigure credentials, environment, or host setup: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1807

4. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail
  • User impact: Developers may misconfigure credentials, environment, or host setup: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail. Context: Observed when using python
  • Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1269

5. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: [Connector Request] Meta Threads Integration for Drafting, Scheduling, and Publishing
  • User impact: Developers may misconfigure credentials, environment, or host setup: [Connector Request] Meta Threads Integration for Drafting, Scheduling, and Publishing
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Connector Request] Meta Threads Integration for Drafting, Scheduling, and Publishing. Context: Observed when using python, docker
  • Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1801

6. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: [Connector Request] neo4j as a source would also be helpful
  • User impact: Developers may misconfigure credentials, environment, or host setup: [Connector Request] neo4j as a source would also be helpful
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Connector Request] neo4j as a source would also be helpful. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1117

7. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: [connector support] push
  • User impact: Developers may misconfigure credentials, environment, or host setup: [connector support] push
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [connector support] push. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1148

8. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: v0.9.64
  • User impact: Upgrade or migration may change expected behavior: v0.9.64
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v0.9.64. Context: Observed when using node
  • Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.64

9. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: v0.9.66
  • User impact: Upgrade or migration may change expected behavior: v0.9.66
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v0.9.66. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.66

10. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: v0.9.69
  • User impact: Upgrade or migration may change expected behavior: v0.9.69
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v0.9.69. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.69

11. Configuration risk: Configuration risk requires verification

  • Severity: medium
  • Finding: Developers should check this configuration risk before relying on the project: v0.9.73
  • User impact: Upgrade or migration may change expected behavior: v0.9.73
  • Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v0.9.73. Context: Source discussion did not expose a precise runtime context.
  • Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.73

12. Capability evidence risk: Capability evidence risk requires verification

  • Severity: medium
  • Finding: README/documentation is current enough for a first validation pass.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: capability.assumptions | https://github.com/airweave-ai/airweave

Source: Doramagic discovery, validation, and Project Pack records

Community Discussion Evidence

These external discussion links are review inputs, not standalone proof that the project is production-ready.

Sources 12

Count of project-level external discussion links exposed on this manual page.

Use Review before install

Open the linked issues or discussions before treating the pack as ready for your environment.

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using airweave with real data or production workflows.

  • [[connector support] push](https://github.com/airweave-ai/airweave/issues/1148) - github / github_issue
  • [[Connector Request] google sheet](https://github.com/airweave-ai/airweave/issues/1151) - github / github_issue
  • Jira connector: enrich issues with comments and basic metadata - github / github_issue
  • [[Connector Request] ERPnext from Frappe](https://github.com/airweave-ai/airweave/issues/1110) - github / github_issue
  • [[Connector Request] neo4j as a source would also be helpful](https://github.com/airweave-ai/airweave/issues/1117) - github / github_issue
  • [[Connector Request] oracle db](https://github.com/airweave-ai/airweave/issues/1130) - github / github_issue
  • [[Connector Request] Obsidian](https://github.com/airweave-ai/airweave/issues/1185) - github / github_issue
  • [[Connector Request] logseq](https://github.com/airweave-ai/airweave/issues/1187) - github / github_issue
  • [[Connector Request] Telegram](https://github.com/airweave-ai/airweave/issues/1188) - github / github_issue
  • [Wordpress[Connector Request]](https://github.com/airweave-ai/airweave/issues/1190) - github / github_issue
  • Security or permission risk requires verification - GitHub / issue
  • Configuration risk requires verification - GitHub / issue

Source: Project Pack community evidence and pitfall evidence