# https://github.com/airweave-ai/airweave Project Manual Generated at: 2026-07-25 07:32:52 UTC ## Table of Contents - [Repository Overview & System Architecture](#page-overview) - [Source Connectors, Entities & Sync Pipeline](#page-connectors-sync) - [Search, Embeddings, Reranking & Retrieval Layer](#page-search) - [Authentication Providers, Billing, MCP Server, Connect Widget & Deployment](#page-platform) ## Repository Overview & System Architecture ### Related Pages Related topics: [Source Connectors, Entities & Sync Pipeline](#page-connectors-sync), [Search, Embeddings, Reranking & Retrieval Layer](#page-search), [Authentication Providers, Billing, MCP Server, Connect Widget & Deployment](#page-platform)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/airweave-ai/airweave/blob/main/README.md) - [backend/airweave/main.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/main.py) - [backend/airweave/architecture-refactor.md](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/architecture-refactor.md) - [backend/airweave/core/container/container.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/core/container/container.py) - [backend/airweave/core/config/settings.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/core/config/settings.py) - [backend/airweave/platform/sync.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/sync.py) - [backend/airweave/platform/entities/base.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/entities/base.py) - [backend/airweave/platform/destinations/vespa.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/destinations/vespa.py)
# Repository Overview & System Architecture ## Purpose and Scope Airweave is an open-source data pipeline platform that ingests content from external SaaS sources, normalizes it into searchable entities, and writes them into a vector-capable destination for downstream AI agents. The system is built to expose those ingested records through the Model Context Protocol (MCP), letting agents query collections in a uniform way regardless of where the underlying data originated. The product surface spans two halves: a Python/FastAPI backend that owns the sync engine, authentication, persistence, and connector orchestration, and a frontend workspace for managing collections, sources, and destinations. The backend lives under `backend/airweave` and is launched via `backend/airweave/main.py`, which wires the FastAPI application, mounts routers, configures middleware, and registers startup hooks for background services. `Source: [backend/airweave/main.py:1-80]()` At the application level, the product concepts a developer needs to recognize are: - **Source** — a connected third-party account (e.g., OneDrive, SharePoint, Google Drive, Notion, MongoDB, custom code). - **Collection** — a logical grouping of sources that share a destination and access policy. - **Destination** — the storage backend where entities land (Vespa is the primary vector destination). - **Sync** — a scheduled or on-demand run that pulls entities from one or more sources and pushes them to the destination. - **Entity** — the normalized schema produced by a connector before it is embedded and indexed. The boundaries of the repository are intentionally wide on the connector side and narrow on the agent side: Airweave does not embed AI itself — it prepares data for agents and exposes it through MCP. ## High-Level Architecture The backend is decomposed into four cooperating layers, each with a clear responsibility and a narrow interface to its neighbors. | Layer | Module path | Responsibility | |-------|-------------|----------------| | API | `backend/airweave/api/` | HTTP/MCP surface, request validation, auth | | Platform / core | `backend/airweave/platform/` | Connectors, entities, sync orchestration, destinations | | Core infrastructure | `backend/airweave/core/` | Settings, dependency container, persistence, queues | | Models & schemas | `backend/airweave/models/`, `schemas/` | ORM and Pydantic shapes shared across layers | The architecture document `backend/airweave/architecture-refactor.md` codifies the layering rules and the way the platform depends on core but never the reverse. The `Container` in `backend/airweave/core/container/container.py` is the composition root: it builds singletons for the database engine, Redis client, temporal client, OAuth providers, and platform registries, then hands them out via FastAPI dependencies. `Source: [backend/airweave/core/container/container.py:1-120]()` Settings are loaded once from environment variables and `.env` files through Pydantic settings in `backend/airweave/core/config/settings.py`, with explicit sections for database, Redis, Temporal, OAuth, embeddings, and feature flags. `Source: [backend/airweave/core/config/settings.py:1-100]()` ## Sync Pipeline The sync pipeline is the operational heart of Airweave. Each step is an isolated unit with a single output contract so it can be retried, replayed, and inspected independently. 1. **Source enumeration** — the connector's `generate_entities` yields entity records, handling pagination, auth refresh, and rate limits. `Source: [backend/airweave/platform/sync.py:1-150]()` 2. **Transformation** — entities pass through user-configurable mapping and chunking stages that produce embeddable chunks. 3. **Embedding** — chunks are vectorized through a configurable embedding backend (OpenAI, Mistral, or self-hosted) — see PR #1803 that made the Mistral base URL configurable. `Source: [backend/airweave/core/config/settings.py:1-100]()` 4. **Destination write** — chunks and vectors are pushed to the destination; the Vespa destination implementation handles indexing, deduplication, and ACL enforcement. `Source: [backend/airweave/platform/destinations/vespa.py:1-120]()` ```mermaid flowchart LR A[Source Connector] --> B[Entity Yield] B --> C[Transform & Chunk] C --> D[Embedding Service] D --> E[Destination - Vespa] E --> F[MCP / Agents] F --> G[Search & Retrieval] ``` Syncs are durable: progress, cursor state, and per-record errors are persisted so a failed sync can be resumed without re-fetching from the source. ArfReplaySource is provided for replaying captured payloads in tests, accepting pipeline kwargs to mirror production stages. `Source: [backend/airweave/platform/sync.py:1-150]()` ## Connectors, Entities, and Destinations Every connector implements a common interface declared on `Entity`, defined in `backend/airweave/platform/entities/base.py`, which guarantees a stable `entity_id`, `webpage_url`, `breadcrumbs`, and `metadata` envelope on every record. This contract is what lets downstream stages and the MCP surface treat OneDrive, SharePoint, Teams, Notion, Appwrite, and custom sources interchangeably. `Source: [backend/airweave/platform/entities/base.py:1-80]()` Recent platform work tightened behavior around edge cases that surface across many connectors: - **Microsoft Graph sources** (OneDrive, SharePoint, Teams) honor Purview sensitivity labels to skip restricted files and sites (#1790), and team-scope sharing links are attached per-file rather than as a blanket grant (#1774, #1775) — both addressing security-sensitive behavior flagged in #1807 where prefetching continued after account lockout. - **Teams messages** must be sanitized before ingestion: the escape character `0x1B` is rejected by Vespa, so connectors strip control codes before yielding entities (#1269). - **Custom connectors** are a long-standing ask (#1268); the platform's `BaseConnector` shape is designed to be subclassed from outside the package, but no first-class registration API is shipped yet. ## Operational Concerns Airweave is designed for self-hosted and cloud deployments with the same code path. The container performs startup checks for the database, Redis, and Temporal endpoints before accepting traffic, and the settings layer exposes a single `disabled` flag surface for optional subsystems like the tabular collection browse view (#1780). For high-availability Redis, Sentinel topologies are recognized (#1796), and workflow failures caused by user-managed credential errors are explicitly excluded from run-level failure counts (#1787), so misconfigured OAuth on a customer side does not poison the operator dashboard. OAuth responses are normalized in the SDK: regardless of whether a provider returns `expires_in` (OAuth 2 standard) or `expires_at` (#1024), the platform persists a unified absolute expiry on the connection record. Configurable API keys per provider (#41) are surfaced through the same settings module, allowing operators to swap OpenAI, Groq, Ollama, or Mistral without code changes. --- ## Source Connectors, Entities & Sync Pipeline ### Related Pages Related topics: [Repository Overview & System Architecture](#page-overview), [Search, Embeddings, Reranking & Retrieval Layer](#page-search)
Related Source Files The following source files were used to generate this page: - [backend/airweave/platform/sources/_base.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/sources/_base.py) - [backend/airweave/platform/entities/_base.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/entities/_base.py) - [backend/airweave/platform/destinations/vespa/destination.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/destinations/vespa/destination.py) - [backend/airweave/domains/sync_pipeline/orchestrator.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/sync_pipeline/orchestrator.py) - [backend/airweave/domains/syncs/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/syncs/service.py) - [backend/airweave/domains/temporal/workflows/run_source_connection.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/temporal/workflows/run_source_connection.py) - [backend/airweave/platform/sources/sharepoint_online.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/sources/sharepoint_online.py) - [backend/airweave/platform/entities/teams_message.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/platform/entities/teams_message.py)
# Source Connectors, Entities & Sync Pipeline The Airweave platform is built around three collaborating subsystems: **Source Connectors** (which pull raw data from external SaaS APIs, databases, and file systems), **Entities** (the normalized in-memory representation of that data), and the **Sync Pipeline** (the orchestrator that streams entities from a source through transformation stages into a destination such as Vespa). A sync run is a single Temporal workflow invocation that drives a connector, consumes its entity stream, and writes documents to the configured destination. ## Source Connector Framework Every connector inherits from a common base class that enforces a uniform async-stream contract, so a new integration only needs to implement data fetching logic rather than the surrounding plumbing. `Source.generate_entities()` is the central entry point; it yields `BaseEntity` instances (or `AsyncIterator[BaseEntity]`) so the rest of the pipeline can stay backend-agnostic. Source: [backend/airweave/platform/sources/_base.py](). Connectors manage their own authentication state. The base class exposes helper methods for OAuth refresh, rate-limit back-off, and cursor persistence; concrete implementations like `SharePointOnlineSource` build on these helpers to deal with Microsoft Graph pagination, sensitivity-label filtering (added in v0.9.70), and per-file sharing-link scoping (v0.9.67). Source: [backend/airweave/platform/sources/sharepoint_online.py](). As reported in issue #1807, connectors must also handle `NEEDS_SOURCE` state correctly when an upstream account is revoked or locked out — the connector should stop prefetching rather than continue to call the API. ## Entity Model Entities are Pydantic-style records that carry both user-visible fields (e.g. message body, file metadata) and framework-level metadata (timestamps, source identifiers, web IDs). The base class `BaseEntity` defines the validation hooks, serialization rules, and embedding-friendly field typing that every concrete entity inherits. Source: [backend/airweave/platform/entities/_base.py](). A concrete example is `TeamsMessageEntity`, which models a Microsoft Teams chat message. Because the destination layer expects strictly UTF-8 text, any non-printable control character (such as `0x1B` ESC) must be sanitized before the entity reaches Vespa — issue #1269 documents a sync failure caused by an unfiltered ESC byte in a Teams message body. Source: [backend/airweave/platform/entities/teams_message.py](). ## Sync Pipeline Orchestrator The orchestrator is the conductor of a sync run. It receives a `Sync` record, resolves the configured source and destination pair, opens a Temporal workflow (`RunSourceConnectionWorkflow`), and threads entities from the source stream through transformation, deduplication, embedding, and write phases. Source: [backend/airweave/platform/destinations/vespa/destination.py]() and [backend/airweave/domains/sync_pipeline/orchestrator.py](). The Temporal workflow is responsible for the long-running mechanics: heartbeat reporting, retry policy, graceful cancellation, and final status emission back to the syncs domain. Source: [backend/airweave/domains/temporal/workflows/run_source_connection.py](). ```mermaid flowchart LR A[Source Connector
generate_entities] --> B[Entity Stream
BaseEntity] B --> C[Transformer / Sanitizer] C --> D[Embedding Stage] D --> E[Destination
e.g. Vespa] E --> F[Sync Record Update] F --> G[Temporal Workflow
RunSourceConnectionWorkflow] ``` The `SyncService` layer sits above the orchestrator and exposes the HTTP API used by the front end. It validates user input, persists the sync configuration, and dispatches a workflow per run; it also normalizes sync outcomes so that customer-side credential errors are not miscounted as workflow failures (fix shipped in v0.9.69). Source: [backend/airweave/domains/syncs/service.py](). ## Lifecycle, Resilience, and Community-Driven Fixes A sync moves through several states — pending, running, completed, failed, or `NEEDS_SOURCE` — that are persisted alongside the workflow execution. Recent releases have tightened this lifecycle: - **v0.9.67** patched SharePoint Online to scope sharing-link viewers per file and stop organization-scoped links from bypassing access control. Source: [release notes](https://github.com/airweave-ai/airweave/releases/tag/v0.9.67). - **v0.9.69** stopped treating customer credential errors as workflow failures, so dashboards no longer misreport transient auth issues. Source: [release notes](https://github.com/airweave-ai/airweave/releases/tag/v0.9.69). - **v0.9.70** added Purview sensitivity-label filtering for SharePoint and OneDrive, allowing organizations to exclude confidential files from indexing. Source: [release notes](https://github.com/airweave-ai/airweave/releases/tag/v0.9.70). - **v0.9.71** accepted sync-pipeline keyword arguments in `ArfReplaySource.generate_entities`, improving replay determinism. Source: [release notes](https://github.com/airweave-ai/airweave/releases/tag/v0.9.71). Community requests such as #41 (configurable AI provider keys), #943 (MongoDB connector), #1024 (OAuth `expires_at`/`expires_in` consistency), and ongoing custom-connector demand (#1268) all shape how this pipeline is expected to evolve toward pluggability and per-tenant credentials. When authoring a new connector, developers should follow the existing base class contract, sanitize text payloads before they reach the destination, and ensure that revoked-access conditions surface as `NEEDS_SOURCE` rather than silent retries. --- ## Search, Embeddings, Reranking & Retrieval Layer ### Related Pages Related topics: [Repository Overview & System Architecture](#page-overview), [Source Connectors, Entities & Sync Pipeline](#page-connectors-sync)
Related Source Files The following source files were used to generate this page: - [backend/airweave/search/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/search/service.py) - [backend/airweave/search/orchestrator.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/search/orchestrator.py) - [backend/airweave/domains/search/agentic/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/search/agentic/service.py) - [backend/airweave/domains/search/classic/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/search/classic/service.py) - [backend/airweave/domains/search/instant/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/search/instant/service.py) - [backend/airweave/domains/embedders/registry.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/embedders/registry.py)
# Search, Embeddings, Reranking & Retrieval Layer ## Overview The Search, Embeddings, Reranking & Retrieval Layer is the read-side counterpart to Airweave's sync pipeline. While the sync layer is responsible for ingesting content from external sources into a vector backend (e.g. Vespa), the retrieval layer answers user/agent queries by composing embeddings, hybrid search, and reranking against that store. It is implemented as a thin service facade (`backend/airweave/search/service.py`) that delegates orchestration to a dedicated component (`backend/airweave/search/orchestrator.py`) and routes requests to one of three domain services: agentic, classic, or instant. Source: [backend/airweave/search/service.py:1-80](). ## Search Service Architecture ### Entry point and orchestration The top-level `SearchService` (in `backend/airweave/search/service.py`) accepts a `SearchRequest`, resolves the target collection, and dispatches to the `SearchOrchestrator`. The orchestrator is responsible for cross-cutting concerns: resolving embedders, applying access-control filters, choosing the correct destination backend, and deciding whether reranking should run. It also performs query expansion for the agentic flow. Source: [backend/airweave/search/orchestrator.py:1-120](). The orchestrator hands the prepared request off to a domain-specific service. This split — a generic orchestrator plus per-mode services — lets Airweave evolve each retrieval style independently without leaking complexity into the public API. Source: [backend/airweave/search/orchestrator.py:120-260](). ### Request → response flow ```mermaid flowchart LR A[Client / Agent] --> B[SearchService] B --> C[SearchOrchestrator] C --> D{Which mode?} D -- agentic --> E[AgenticService] D -- classic --> F[ClassicService] D -- instant --> G[InstantService] E --> H[EmbedderRegistry] F --> H G --> H H --> I[Vector Destination] I --> J[Optional Reranker] J --> K[SearchResponse] ``` ## Search Modes ### Agentic search The agentic service (`backend/airweave/domains/search/agentic/service.py`) is intended for tool-calling agents exposed over MCP. It returns structured results (documents + optional generated answers) and supports iterative, multi-step retrieval: the agent can call back with refined queries until it has enough evidence. The service produces both a list of retrieved entities and a natural-language answer synthesized from the top results. Source: [backend/airweave/domains/search/agentic/service.py:1-140](). ### Classic search The classic service (`backend/airweave/domains/search/classic/service.py`) is a single-shot, request/response retrieval. It performs dense (vector) and, when the destination supports it, lexical (BM25-style) retrieval, merges them, optionally reranks, and returns a ranked list of documents. This is the mode used by the public REST `/search` endpoint and by the dashboard UI. Source: [backend/airweave/domains/search/classic/service.py:1-180](). ### Instant search The instant service (`backend/airweave/domains/search/instant/service.py`) optimizes for low-latency typeahead. It issues a lightweight vector query with a small `top_k`, skips heavy reranking, and is designed to fire on every keystroke from the frontend search box. It shares the same orchestrator and embedder registry but bypasses post-processing that would add latency. Source: [backend/airweave/domains/search/instant/service.py:1-110](). ## Embedding Layer ### Embedder registry All three modes pull their embedding model through a single registry (`backend/airweave/domains/embedders/registry.py`). The registry maps a logical model name (e.g. `openai/text-embedding-3-small`, `cohere/embed-english-v3.0`, a local sentence-transformer) to a concrete client implementation, handling API key resolution, dimension metadata, and batching. Because the embedder is resolved per-request via the orchestrator, a single Airweave deployment can mix embedders across collections without code changes. Source: [backend/airweave/domains/embedders/registry.py:1-160](). ### Reranking Reranking is a post-retrieval step orchestrated by `SearchOrchestrator` rather than embedded in any single mode. After the destination returns its initial candidates, the orchestrator may invoke a reranker (Cohere, a cross-encoder, or a no-op when disabled by configuration) to reorder the top-N results before returning them to the caller. Source: [backend/airweave/search/orchestrator.py:200-320](). ## Configuration and Extensibility The retrieval layer is designed to be configurable per collection: each collection stores its embedder choice and destination backend, and the orchestrator reads these to wire up the request. Because embedders, destinations, and rerankers are all addressed through registries rather than hard-coded imports, new providers can be added by registering a client — without modifying the search services themselves. This is the same pattern the sync pipeline uses on the write side, keeping the read and write paths symmetrical. Source: [backend/airweave/search/orchestrator.py:80-180](). ## Community-Relevant Notes Several community discussions relate directly to this layer. Configurable API keys for OpenAI and other providers (issue #41) map onto the embedder registry's responsibility for resolving credentials. OAuth `expires_in` vs `expires_at` (issue #1024) affects the embedder and destination clients that hold short-lived tokens. Connector requests (Obsidian, Firecrawl, MongoDB, Appwrite, Odoo, Meta Threads) primarily expand the *write* side, but every new connector depends on the retrieval layer to expose its data, so the embedder and orchestrator choices made here directly affect what newly-synced content can be queried. Source: [backend/airweave/domains/embedders/registry.py:160-260](). --- ## Authentication Providers, Billing, MCP Server, Connect Widget & Deployment ### Related Pages Related topics: [Repository Overview & System Architecture](#page-overview), [Source Connectors, Entities & Sync Pipeline](#page-connectors-sync)
Related Source Files The following source files were used to generate this page: - [backend/airweave/domains/auth_provider/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/auth_provider/service.py) - [backend/airweave/domains/auth_provider/schemas.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/auth_provider/schemas.py) - [backend/airweave/domains/oauth/oauth2_service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/oauth/oauth2_service.py) - [backend/airweave/domains/oauth/router.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/oauth/router.py) - [backend/airweave/domains/billing/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/billing/service.py) - [backend/airweave/domains/billing/router.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/billing/router.py) - [backend/airweave/domains/usage/ledger.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/usage/ledger.py) - [backend/airweave/domains/source_connections/service.py](https://github.com/airweave-ai/airweave/blob/main/backend/airweave/domains/source_connections/service.py) - [mcp/src/server.ts](https://github.com/airweave-ai/airweave/blob/main/mcp/src/server.ts) - [mcp/src/tools.ts](https://github.com/airweave-ai/airweave/blob/main/mcp/src/tools.ts) - [frontend/src/components/ConnectWidget/index.tsx](https://github.com/airweave-ai/airweave/blob/main/frontend/src/components/ConnectWidget/index.tsx) - [docker/docker-compose.yml](https://github.com/airweave-ai/airweave/blob/main/docker/docker-compose.yml)
# Authentication Providers, Billing, MCP Server, Connect Widget & Deployment ## 1. Purpose and Scope This page covers the cross-cutting surface of Airweave that sits between data sources and end-user applications. It documents the four cooperating subsystems: - **Authentication Providers** – pluggable backend logic that brokers OAuth handshakes and stores per-source credentials. - **Billing** – subscription state, plan limits, and the usage ledger that meters sync operations. - **MCP Server** – the Model Context Protocol (MCP) entrypoint that lets external agents (e.g., Claude, Antigravity) invoke Airweave tools. - **Connect Widget & Deployment** – the embeddable frontend component that drives the user-facing OAuth flow, plus the containerized topology that ships it. Together they form the path a user follows: open the **Connect Widget**, complete an OAuth flow managed by an **Auth Provider**, store the resulting **Source Connection**, meter the resulting activity in the **Billing/Usage** domain, and expose the searchable corpus to an agent through the **MCP Server**. ## 2. Authentication Providers and OAuth Flow Airweave treats authentication as a domain-first concern, isolating OAuth protocol code from connector logic. - `backend/airweave/domains/auth_provider/service.py` defines the abstract `AuthProviderService` interface. Concrete subclasses (`AsanaAuthProvider`, `GithubAuthProvider`, `SharepointAuthProvider`, etc.) implement the `get_auth_url`, `exchange_code_for_token`, and `refresh_token` lifecycle hooks. Source: `backend/airweave/domains/auth_provider/service.py:1-120`. - The provider registry is keyed by short-name (`asana`, `github`, `sharepoint`, …) and resolved through `AuthProviderService.for_source(short_name)`. This allows the connector layer to remain agnostic about OAuth specifics. Source: `backend/airweave/domains/auth_provider/service.py:121-180`. - `backend/airweave/domains/oauth/oauth2_service.py` owns the PKCE/state pair generation, redirect URI validation, and token persistence into the `source_connection` table. It also normalizes the `expires_in` field into the `expires_at` timestamp consumers expect. Source: `backend/airweave/domains/oauth/oauth2_service.py:1-95`. - The HTTP surface is exposed by `backend/airweave/domains/oauth/router.py` with endpoints for `/oauth/{provider}/authorize` and `/oauth/{provider}/callback`. Source: `backend/airweave/domains/oauth/router.py:1-60`. This separation addresses the community pain point tracked in issue #1024, where SDK users hit inconsistencies between `expires_at` and `expires_in`; the normalization happens centrally in `oauth2_service.py` rather than at every connector. ## 3. Billing and Usage Ledger The billing domain is read-mostly at request time and write-mostly at sync completion. - `backend/airweave/domains/billing/service.py` wraps the Stripe-compatible customer, subscription, and invoice objects, translating plan IDs into in-app quota strings. Source: `backend/airweave/domains/billing/service.py:1-140`. - `backend/airweave/domains/billing/router.py` exposes `/billing/portal`, `/billing/webhook`, and `/billing/quota` for the dashboard and for Stripe webhooks. Source: `backend/airweave/domains/billing/router.py:1-70`. - `backend/airweave/domains/usage/ledger.py` records every successful entity ingestion as a ledger entry keyed by `(organization_id, source_connection_id, day)`. Entries roll up into the per-organization quota reported by `BillingService.remaining_quota`. Source: `backend/airweave/domains/usage/ledger.py:1-110`. - Source-connection lifecycle hooks in `backend/airweave/domains/source_connections/service.py` call `ledger.record()` after each successful sync run, and surface a `NEEDS_SOURCE` state when the stored credential is rejected (relevant to issue #1807 about OneDrive prefetch after account lockout). Source: `backend/airweave/domains/source_connections/service.py:1-160`. ## 4. MCP Server and Connect Widget The two halves of the user surface are deliberately thin clients over the HTTP API. - `mcp/src/server.ts` boots an MCP server using the `@modelcontextprotocol/sdk`, registers tool handlers, and proxies authenticated requests to the Airweave backend using a per-tenant API key. Source: `mcp/src/server.ts:1-95`. - `mcp/src/tools.ts` enumerates the tools exposed to MCP clients (`search`, `list_collections`, `trigger_sync`, `get_source_status`). Each tool maps 1:1 to a backend REST endpoint. Source: `mcp/src/tools.ts:1-80`. - `frontend/src/components/ConnectWidget/index.tsx` is the embeddable React component that a host application mounts inside its own DOM tree. It opens a popup, walks the user through the `/oauth/{provider}/authorize → /callback` round trip, and finally POSTs to `/source-connections` with the returned `code`. Source: `frontend/src/components/ConnectWidget/index.tsx:1-130`. - The widget exposes a `onSuccess(connection)` callback so host apps can navigate to their next step without re-fetching state. ## 5. Deployment Topology Airweave ships as a multi-service stack orchestrated by Docker Compose, with optional Kubernetes manifests for production. - `docker/docker-compose.yml` defines services for the FastAPI backend, the Postgres database, Redis (with Sentinel support added in v0.9.72), Vespa for vector search, Temporal for workflows, and the MCP server. Source: `docker/docker-compose.yml:1-90`. - Environment configuration flows through `.env` files consumed by Pydantic settings; secrets (OAuth client secrets, Stripe keys, Mistral base URL added in v0.9.73) are never baked into images. - The MCP container is stateless; it pulls per-tenant keys at startup from `AIRWEAVE_API_URL` and `AIRWEAVE_API_KEY`, making horizontal scaling safe. ## 6. End-to-End Flow ```mermaid sequenceDiagram participant U as User participant CW as Connect Widget participant API as Airweave API participant AP as Auth Provider participant LED as Usage Ledger participant MCP as MCP Server U->>CW: Click "Connect Asana" CW->>API: GET /oauth/asana/authorize API->>AP: get_auth_url(state, PKCE) AP-->>U: Redirect to Asana login U->>API: /oauth/asana/callback?code=... API->>AP: exchange_code_for_token(code) API->>LED: record source_connection.create MCP->>API: tools/call search({q}) API-->>MCP: results + quota snapshot ``` This diagram ties the four subsystems together: the widget drives OAuth, the auth provider persists credentials, the ledger meters activity for billing, and the MCP server exposes the resulting corpus to agents. --- --- ## Pitfall Log Project: airweave-ai/airweave Summary: Found 26 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Security or permission risk - Security or permission risk requires verification. ## 1. Security or permission risk - Security or permission risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/airweave-ai/airweave/issues/1801 ## 2. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.host_targets | https://github.com/airweave-ai/airweave ## 3. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE - User impact: Developers may misconfigure credentials, environment, or host setup: OneDrive connector appears to continue prefetching after Microsoft account lockout / source state NEEDS_SOURCE - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1807 ## 4. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail - User impact: Developers may misconfigure credentials, environment, or host setup: [Bug] Sync Failure: Illegal code point 0x1B in TeamsMessageEntity causes Vespa ingestion to fail - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1269 ## 5. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: [Connector Request] Meta Threads Integration for Drafting, Scheduling, and Publishing - User impact: Developers may misconfigure credentials, environment, or host setup: [Connector Request] Meta Threads Integration for Drafting, Scheduling, and Publishing - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1801 ## 6. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: [Connector Request] neo4j as a source would also be helpful - User impact: Developers may misconfigure credentials, environment, or host setup: [Connector Request] neo4j as a source would also be helpful - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1117 ## 7. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: [connector support] push - User impact: Developers may misconfigure credentials, environment, or host setup: [connector support] push - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1148 ## 8. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: v0.9.64 - User impact: Upgrade or migration may change expected behavior: v0.9.64 - Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.64 ## 9. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: v0.9.66 - User impact: Upgrade or migration may change expected behavior: v0.9.66 - Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.66 ## 10. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: v0.9.69 - User impact: Upgrade or migration may change expected behavior: v0.9.69 - Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.69 ## 11. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: v0.9.73 - User impact: Upgrade or migration may change expected behavior: v0.9.73 - Evidence: failure_mode_cluster:github_release | https://github.com/airweave-ai/airweave/releases/tag/v0.9.73 ## 12. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/airweave-ai/airweave ## 13. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/airweave-ai/airweave ## 14. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/airweave-ai/airweave ## 15. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/airweave-ai/airweave ## 16. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/airweave-ai/airweave/issues/1807 ## 17. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/airweave-ai/airweave/issues/1269 ## 18. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: Wordpress[Connector Request] - User impact: Developers may hit a documented source-backed failure mode: Wordpress[Connector Request] - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1190 ## 19. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] ERPnext from Frappe - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] ERPnext from Frappe - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1110 ## 20. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] Obsidian - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] Obsidian - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1185 ## 21. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] Telegram - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] Telegram - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1188 ## 22. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] google sheet - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] google sheet - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1151 ## 23. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] logseq - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] logseq - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1187 ## 24. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [Connector Request] oracle db - User impact: Developers may hit a documented source-backed failure mode: [Connector Request] oracle db - Evidence: failure_mode_cluster:github_issue | https://github.com/airweave-ai/airweave/issues/1130 ## 25. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/airweave-ai/airweave ## 26. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/airweave-ai/airweave