Related Pages

Related topics: System Architecture & Tech Stack, MRP Pipeline: Document Ingestion & Wiki Compilation

Introduction & Key Features

Arkon is an open-source knowledge management platform that converts uploaded company documents into structured, searchable wikis with the help of large language models. It is designed for internal use within organizations where documents (PDFs, Word files, slides, etc.) accumulate across departments and need to be consolidated into a single, queryable knowledge base.

The project targets teams that want an AI-augmented internal wiki without surrendering their data to public SaaS. The release notes describe the goal as "a significant architectural simplification" consolidating "data and security boundaries" so that workspaces, documents, and access controls all live in one cohesive model (Source: CHANGELOG.md:1-30).

What Arkon Does

At a high level, Arkon performs three jobs:

1. Ingest — Users upload documents into a workspace.
2. Synthesize — An LLM (Claude or Gemini, configured per deployment) extracts entities, summaries, and topic groupings from the extracted text.
3. Publish — The synthesized content is rendered as a navigable wiki, browsable by users in the workspace.

The system is composed of a FastAPI backend, a PostgreSQL database (accessed through asyncpg), a frontend, and an MCP (Model Context Protocol) server for tool integration with AI assistants (Source: docker-compose.yml:1-60).

Key Features

Document Upload & Indexing Arkon accepts files via the API, extracts text, and persists it as Source.full_text. Because PostgreSQL text/varchar and JSON columns cannot store the NUL byte (0x00), certain binary-tainted extractions have caused CharacterNotInRepertoireError during indexing (Source: app/database/models.py:1-50; reported in community issue #21).

AI-Generated Wiki Pages The /api/wiki/pages endpoint creates wiki pages from ingested sources. Page creation relies on a hybrid page_type property on the WikiPage model. A missing setter caused 500 responses on POST until it was patched (Source: app/routers/wiki.py:1-80; reported in community issue #20).

Workspace-Based Organization As of v0.9.0, the legacy "Project" silo layer was removed. Workspaces are now the top-level boundary for both data and security, simplifying membership management. Role values for members were also corrected from a hard-coded "member" to selectable roles such as "viewer" (Source: CHANGELOG.md:1-30; community issue #4).

Scope-Based Access Control Access to wikis and documents can be partitioned by department ("scope"). The scopes router exposes endpoints such as /api/scopes/* and works with models Scope, ScopeMembership, and ScopeRole. A regression where the router was not registered in app/main.py produced 404s on every scope endpoint, which has since been fixed (Source: app/main.py:1-80, app/routers/scopes.py:1-50; community issue #8, #10, #11).

MCP Server Integration Arkon ships an MCP server that lets Claude (or compatible agents) query the local knowledge base. Configuration uses the standard mcpServers JSON block, e.g. via mcp-remote against a local HTTP endpoint (Source: README.md:1-120; community issue #7).

Architecture at a Glance

The backend boots all routers inside app/main.py; missing router registrations are a common source of 404s after refactors (Source: app/main.py:1-80).

Deployment & Known Limitations

Arkon is distributed with a docker-compose.yml and a .env.docker template. A common first-run failure is "wrong password" at login — the API expects DATABASE_URL values that match the POSTGRES_* block (Source: docker-compose.yml:1-60, README.md:1-120; community issue #6).

Reported limitations worth noting before adoption:

• LLM provider lock-in. Today only Claude and Gemini are supported; self-hosted models such as Gemma or Qwen are a requested enhancement (community issue #19).
• No wiki language selector. Uploads of mixed-language sources produce mixed-language wiki pages with no UI control to pin a primary language (community issue #15).
• Source attribution. Wiki pages originally aggregated all sources without a clear visual cue indicating which source a passage came from; improvements have landed but attribution density remains a usability focus (community issue #9).
• Skill/IP protection. Operators sharing Claude skill sets inside Arkon have raised concerns about employees extracting and reusing proprietary skills after leaving — there is no built-in DRM layer yet (community issue #5).

For a deeper view of the workspace/scope data model and the rationale behind the v0.9.0 simplification, refer to DESIGN.md (Source: DESIGN.md:1-80).

Related Pages

Related topics: Introduction & Key Features, Deployment, Configuration & Troubleshooting

System Architecture & Tech Stack

Overview

Arkon is a document-centric knowledge platform that ingests uploaded files, extracts structured content, and synthesizes them into navigable wikis. The system is delivered as a containerized Python backend with an asynchronous job runner, a relational metadata store, and a separate web frontend. As of v0.9.0 the legacy "Project" silo has been removed and the data model is consolidated around Workspaces, Scopes (departmental access boundaries), Documents/Sources, and Wiki Pages Source: docs/ARCHITECTURE.md:1-40. The platform relies on external LLM providers (Claude, Gemini) for synthesis and requires outbound network connectivity to those services Source: requirements.txt:1-60.

High-Level Architecture

Arkon follows a three-tier, service-oriented layout: an HTTP API tier, a background worker tier, and a persistence tier, all orchestrated by Docker Compose for local and self-hosted deployments Source: docker-compose.yml:1-80.

flowchart LR
  Browser[Web Frontend]
  API[FastAPI App<br/>app/main.py]
  Worker[Background Worker<br/>app/worker.py]
  DB[(PostgreSQL)]
  FS[(Object / File Storage)]
  LLM[Claude / Gemini API]
  Browser -- REST/JSON --> API
  API --> DB
  API --> Worker
  Worker --> DB
  Worker --> FS
  Worker --> LLM
  API --> LLM

The API process owns request/response handling, authentication, authorization, and CRUD over the metadata schema Source: app/main.py:1-120. Long-running or CPU/IO-heavy work — text extraction, chunking, embedding, and LLM synthesis — is offloaded to a dedicated worker that consumes queued jobs and writes results back to the same database Source: app/worker.py:1-90. This separation keeps request latency predictable and isolates third-party API failures from interactive flows.

Backend Stack

The backend is written in Python and built on FastAPI with fully asynchronous route handlers, as evidenced by the asyncpg-based async engine configuration Source: app/database/models.py:1-40. Key dependencies include:

• FastAPI + Starlette for the HTTP layer and OpenAPI generation Source: requirements.txt:1-30.
• SQLAlchemy 2.x (async) as the ORM, with hybrid_property descriptors used for derived columns such as page_type Source: app/database/models.py:120-180.
• Alembic for schema migrations, configured through alembic/env.py and sequenced in the 031–036 revision range after the v0.9.0 refactor Source: alembic/env.py:1-80.
• Pydantic models for request/response validation, consistent with FastAPI conventions.
• An HTTP client for outbound calls to LLM providers (Claude, Gemini) and optional MCP skill servers Source: requirements.txt:30-80.

Routers are modular — one module per bounded context (scopes, wiki, documents, auth, …) — and must be explicitly registered on the FastAPI() instance in app/main.py. Missing registrations have caused real outages: in commit 1198a5b, the scopes router was wired up only after /api/scopes/* returned 404s on a fresh clone Source: app/main.py:80-160 Source: app/api/scopes.py:1-60.

Data Layer

PostgreSQL is the single source of truth for metadata and, after extraction, for the full text of indexed sources Source: docker-compose.yml:20-60. The connection string follows the postgresql+asyncpg://<user>:<pass>@<host>/<db> pattern; community reports show default credentials such as arkon:arkon_secret being common in shipped .env.docker files Source: docker-compose.yml:40-70.

The schema is migrated via Alembic. After the v0.9.0 consolidation, migrations were renamed and reordered into a 031–036 sequence to reflect the Workspace-only world Source: alembic/env.py:40-100. Notable model behaviors that affect operators:

• Source.full_text is stored as text and cannot contain 0x00; PostgreSQL raises CharacterNotInRepertoireError if the extractor leaks NUL bytes, which currently aborts ingestion Source: app/database/models.py:200-260.
• WikiPage.page_type is exposed as a SQLAlchemy hybrid_property; because no setter is defined, attempting to assign it programmatically raises AttributeError and surfaces as a 500 on POST /api/wiki/pages Source: app/database/models.py:260-310.
• Cross-department access control is modeled via Scope, ScopeMembership, and ScopeRole rows, joined from User and resolved at request time Source: app/api/scopes.py:60-140.

Binary uploads (PDF, DOCX, etc.) are persisted alongside the database — either to a mounted volume or an S3-compatible store — and referenced from the Source row by path Source: app/services/document_indexer.py:1-80.

Asynchronous Processing & Workers

Document indexing is a multi-step pipeline implemented in app/services/document_indexer.py: download/load the file, extract text, sanitize (the 0x00 strip is a known gap — see issue #21), chunk, embed, and persist to Source.full_text plus auxiliary tables Source: app/services/document_indexer.py:80-200. Because these steps are slow and may invoke external LLM APIs, they run inside app/worker.py rather than inside the request coroutine Source: app/worker.py:1-90.

Wiki synthesis — turning extracted sources into structured pages — follows the same pattern: the API records intent and enqueues a job; the worker drives the LLM call, applies the chosen language (currently the source language; per-issue #15 there is no UI selector yet), and writes WikiPage rows when complete Source: app/api/wiki.py:1-120.

Deployment Topology

docker-compose.yml brings up three core services for production-shaped local runs Source: docker-compose.yml:1-80:

Operational lessons reported by users: ensure DATABASE_URL matches the Postgres credentials exactly (issue #6 was a login failure traceable to mismatched POSTGRES_PASSWORD vs DATABASE_URL password), and always rebuild after router changes so stale containers do not crash-loop on import errors Source: app/main.py:1-80 Source: docker-compose.yml:60-100.

Known Constraints & Forward-Looking Notes

• LLM lock-in: the stack currently requires outbound access to Claude and/or Gemini; self-hosted models (Gemma, Qwen) are not yet pluggable — a frequently requested enhancement tracked in issue #19 Source: requirements.txt:30-80.
• Wiki language selection: the extractor does not expose a "primary language" knob for synthesis, so mixed-language corpora can produce inconsistent pages (issue #15) Source: app/services/document_indexer.py:200-260.
• Skill/IP protection: there is no built-in DRM for shared MCP skill bundles (issue #5); access control today is enforced at the Workspace/Scope boundary, not at the artifact level Source: app/api/scopes.py:140-220.

Related Pages

Related topics: Wiki Browser, Drafts & Branches, Deployment, Configuration & Troubleshooting

MRP Pipeline: Document Ingestion & Wiki Compilation

Overview

The MRP (Map-Reduce-Process) Pipeline is Arkon's backend system that ingests uploaded documents and compiles them into structured Wiki pages. It lives under app/ai/mrp/ and is split into six modules, each handling a discrete phase of the workflow: pipeline.py orchestrates the run, while mapper.py, reducer.py, merger.py, verifier.py, and writer.py perform individual transformations on Source records to eventually produce WikiPage rows. Source: app/ai/mrp/pipeline.py

The pipeline is invoked after a document is uploaded. Its job is to take the raw extracted text, chunk it, summarize each chunk against the workspace context, combine the summaries into a coherent narrative, sanity-check the result, and persist it as browsable wiki content. Community feedback indicates this flow is central to day-to-day usage — most reported issues (issues #9, #15, #20, #21) sit on the boundary between document ingestion and wiki compilation rather than on unrelated subsystems.

Pipeline Stages

The six modules form a strict linear pipeline. Each stage consumes the output of the previous one and hands a structured artifact to the next.

1. Mapper — Chunking and Initial Grouping

mapper.py receives raw Source.full_text payloads (issue #21 confirms this column exists on the Source model) and splits them into smaller, semantically meaningful units suitable for downstream LLM calls. Community evidence shows the mapper is the first stage that touches extracted text, which means sanitization concerns (such as NUL-byte 0x00 removal) surface here. Source: app/ai/mrp/mapper.py

2. Reducer — Per-Chunk Summarization

reducer.py processes each mapped chunk independently and produces a condensed summary. This stage is typically the most LLM-intensive; community issue #19 requests support for self-hosted models (Gemma/Qwen) which would most naturally replace the inference client called from this module. The reducer is also where language detection implicitly happens — issue #15 documents that no user-controllable language selector currently exists, so each chunk is summarized in whatever language the model chooses. Source: app/ai/mrp/reducer.py

3. Merger — Cross-Source Consolidation

merger.py combines the per-chunk summaries produced by the reducer into a unified document-level narrative. Issue #9 reports that this stage currently pools all Source records together without clearly tagging which summary originated from which document, producing an output where the origin of each paragraph is ambiguous. Source: app/ai/mrp/merger.py

4. Verifier — Consistency and Quality Gates

verifier.py checks the merged output for internal consistency, completeness relative to the source set, and adherence to wiki formatting constraints before the final write. This is a defensive stage; failures here typically surface as 500 responses on POST /api/wiki/pages (issue #20) when structural invariants are violated. Source: app/ai/mrp/verifier.py

5. Writer — Persistence to `WikiPage`

writer.py takes the verified artifact and creates or updates WikiPage rows. Issue #20 traces a 500 here to a missing setter on the page_type hybrid_property, meaning the writer must populate page_type via the underlying mapped column rather than the hybrid attribute. The writer is also where Source.full_text data is finally persisted via the mapper/reducer chain, which is why issue #21 manifests at write time rather than at upload time. Source: app/ai/mrp/writer.py

6. Pipeline — Orchestration

pipeline.py wires the five stages together, manages transactional boundaries, and exposes the entry point used by the document and wiki routers. Source: app/ai/mrp/pipeline.py

Data Flow

flowchart LR
    Upload[Uploaded Document] --> Src[Source.full_text]
    Src --> Map[mapper.py<br/>chunking]
    Map --> Reduce[reducer.py<br/>per-chunk summary]
    Reduce --> Merge[merger.py<br/>cross-source narrative]
    Merge --> Verify[verifier.py<br/>consistency checks]
    Verify --> Write[writer.py<br/>persist WikiPage]
    Write --> Wiki[(Wiki Pages)]

Known Limitations from Community Feedback

These issues cluster around the ingestion boundary and the final write, suggesting the mapper and writer stages are the highest-leverage points for hardening the pipeline. Source: app/ai/mrp/pipeline.py

Related Pages

Related topics: MRP Pipeline: Document Ingestion & Wiki Compilation, Department Scopes, RBAC & Audit

Wiki Browser, Drafts & Branches

The Wiki subsystem in Arkon is split across four FastAPI routers (wiki, wiki_drafts, wiki_branches, wiki_images) and two service-layer modules (wiki_service, contribution_service). Together they expose the read-side browser (listing and rendering pages), the write-side staging area (drafts), the version-control-like parallel tracks (branches), and the media attachments (images) used by wiki content. The split mirrors the same separation seen across the rest of the codebase, where routers are thin HTTP adapters and services own business logic and persistence orchestration. Source: app/routers/wiki.py

Wiki Browser

The wiki router is the primary read endpoint for the wiki surface. It resolves pages by slug, workspace, or hierarchical path and returns the rendered content plus metadata. It is the entry point that the frontend calls when a user opens the wiki sidebar or navigates between pages, and it is also what powers the global search across an organization. Source: app/routers/wiki.py

wiki_service sits underneath the router and centralises page lookups, slug resolution, and the assembly of the rendered payload returned to the client. By keeping all wiki business logic in the service, the router only handles authentication, request validation, and response shaping. This pattern lets the same service be reused by the drafts and branches routers, which need to read existing pages before staging a new edit. Source: app/services/wiki_service.py

A community-reported problem in issue #9 ("Wiki phần wiki đang dồn toàn bộ sources vào gây rối") shows that the browser view must clearly attribute sources to their originating page. The fix referenced in the issue thread scopes sources per-page rather than aggregating them globally, and the wiki router is the layer responsible for enforcing that scoping when serialising a page response. Source: app/routers/wiki.py

Drafts

wiki_drafts provides a staging area for page edits that have not yet been published. A draft is a writeable, isolated copy of a page (or a new page that does not yet exist as a live page) that an author can iterate on without affecting the published wiki. The router exposes endpoints to create, list, read, update, and delete drafts, mirroring the lifecycle of a typical CMS draft. Source: app/routers/wiki_drafts.py

Drafts interact with the live wiki through the publish action, which promotes the draft into a real page. The wiki_service is the natural owner of that promotion step because it already controls page persistence and the rendering pipeline; the drafts router should call into it rather than duplicating the logic. Source: app/services/wiki_service.py

The contribution_service plays a complementary role. While drafts are personal staging, contributions track who contributed which content. The service is what produces the attribution shown in the wiki browser, so when a draft is published the contribution record is finalised at the same time so that history is preserved. Source: app/services/contribution_service.py

A known regression around draft creation is documented in issue #20 ("Creating a wiki page fails with 500 — page_type hybrid_property has no setter"). The failure occurs because the model uses a hybrid_property for page_type and the SQLAlchemy ORM requires an explicit setter to write the value back. Any code path that creates a draft row and assigns page_type triggers the 500 until the setter is added to the model. This is the canonical example of a drafts bug that surfaces in the API layer even though the root cause is in the persistence model. Source: app/routers/wiki_drafts.py

Branches

wiki_branches implements a git-like branch abstraction on top of wiki pages. Each branch is a named track that can hold a divergent version of a page or a set of pages, allowing multiple authors to work in parallel without overwriting one another. Branches differ from drafts in that drafts are short-lived personal staging, while branches are first-class objects that can be reviewed, merged, and shared. Source: app/routers/wiki_branches.py

A typical branch lifecycle is: create branch from a source page → edit pages inside the branch → open a merge request to bring the branch back into the main wiki tree → on approval, the branch router delegates to wiki_service to apply the changes to the live pages and to contribution_service to record the merged authors. This keeps merge logic next to the same services that own normal page writes. Source: app/routers/wiki_branches.py

Branch operations share many data shapes with drafts (both carry a working copy of page content plus metadata), so the wiki service exposes the common read/write primitives that both routers consume. Centralising this avoids drift between the two routers and ensures a merge from a branch produces the same page shape as a draft publish. Source: app/services/wiki_service.py

Images and Cross-cutting Concerns

wiki_images handles uploads and retrieval of media that is embedded in wiki pages. The router stores image binaries and returns the URL that the page renderer substitutes into the markdown. Because images are referenced from both live pages and drafts/branches, the router is intentionally separate so the content routers do not need to deal with multipart upload handling. Source: app/routers/wiki_images.py

The four routers together form a complete edit-publish-browse loop, with wiki_service as the single source of truth for page state and contribution_service as the audit trail for who changed what. This separation also makes the wiki subsystem resilient to the kinds of failures observed in the issue tracker — for example, the page_type setter regression in issue #20 is contained to the persistence layer, and fixing it there restores both draft creation and branch merge paths without changes to the routers themselves. Source: app/services/wiki_service.py

Related Pages

Related topics: Wiki Browser, Drafts & Branches, MCP Server & AI Provider Catalog

Department Scopes, RBAC & Audit

Arkon implements a layered authorization model in which Department Scopes define organizational boundaries (e.g., Product, Sales, FIN), RBAC (Role-Based Access Control) governs the actions a user can perform inside a workspace, and an Audit trail records who did what. Together these three subsystems answer three distinct questions: *which department does the request belong to*, *is the actor allowed to do this*, and *what happened afterwards*.

This page summarizes how those concerns are organized in the codebase and highlights known failure modes drawn from the issue tracker.

Scopes: the Department Boundary

A Scope is the unit that lets administrators partition the workspace into per-department islands of data. Scopes are exposed through the scopes router, mounted at /api/scopes/* once it is registered in app/main.py. The router exposes endpoints to create, list, and assign departments, and it depends on three ORM models: Scope, ScopeMembership, and ScopeRole, plus an Action enum that describes permitted verbs.

A ScopeMembership row links a user to a Scope under a particular ScopeRole, which determines whether the user can read or administer that department's wiki, documents, and skills. Because the Action enum is imported alongside the membership models, the same set of verbs can be reused across routers, keeping authorization vocabulary consistent.

A known regression surfaced in issue #10: after the scopes router was wired into app/main.py, the import of Action, ScopeMembership, and ScopeRole from app.database.models produced undefined names, which crashed the API container on startup. The same root cause had previously produced 404s on /api/scopes/... calls in issue #8 because the router had never been registered. Source: app/routers/scopes.py

Cross-department authorization is the most frequent community question (see issue #11): an administrator wants User A to read the Product and Sales departments while User B only reads FIN. The intended model is to grant multiple ScopeMembership rows per user, one per department, so read access is the union of every membership's role scope. Source: app/routers/scopes.py

RBAC: Role Enforcement inside a Workspace

Where scopes control *which department*, RBAC controls *what role* the user holds inside the workspace they are currently operating on. The router in app/routers/rbac.py exposes the endpoints that list and change workspace roles, and app/services/permissions.py provides the lookup helpers used by the rest of the application to resolve the caller's effective role for a given workspace.

Two services cooperate at request time:

• permission_engine.py evaluates *static* role membership: it asks "does this user have role X on workspace Y?" using the persisted WorkspaceMember rows. It is the layer that defends every router with a simple membership check.
• policy_engine.py evaluates *dynamic* constraints: it composes static membership with contextual rules such as ownership, scope membership, or per-action overrides returned by the Action enum.

Roles exposed to the API use the values viewer, member, and admin tiers. A defect fixed in v0.9.1 caused handleAddMember on the client to hardcode role: "member", which the server rejected for users that should have been viewer (see issue #4). The fix introduced a selectedRole state with viewer as the default and a select dropdown. Source: app/routers/rbac.py

The permission flow can be summarized as follows:

flowchart LR
    A[Request] --> B[Auth: resolve user]
    B --> C{permission_engine<br/>has workspace role?}
    C -- no --> Z[403 Forbidden]
    C -- yes --> D{policy_engine<br/>allows Action on resource?}
    D -- no --> Z
    D -- yes --> E[Handler runs]
    E --> F[audit.log writes event]

Source: app/services/permission_engine.py, app/services/policy_engine.py

Audit: the Recorded Trail

app/routers/audit.py exposes read endpoints over the audit log, while the audit write path is invoked from inside the permission and policy services whenever a privileged action is allowed. The log captures the actor, the resolved scope, the workspace, the action verb (drawn from the same Action enum used by RBAC), and a timestamp.

In practice the audit layer is what makes the difference between "user A tried to view FIN" and "user A was denied". Both transitions are recorded, which lets administrators trace either cross-department reads or denied attempts when investigating incidents such as the skill-leak concern raised in issue #5.

Because the audit router reads the same tables written by the engine, it must be registered alongside scopes and rbac in app/main.py. If a router is missing from the include list, every dependent UI surface degrades silently: the classic symptom was the user-facing message *"Failed to fetch"* masking a 404 on the API call (see issue #8). Source: app/routers/audit.py

Operational Notes and Known Pitfalls

Several recurring defects tie directly back to this subsystem:

• Router not registered. A new router that imports scope or RBAC models must also be added to app/main.py. Skipping this step produces 404s that look like missing data. Source: app/routers/scopes.py
• Import order / undefined names. Action, ScopeMembership, and ScopeRole must be exported from app.database.models before scopes.py is imported. Otherwise Alembic migrations may still apply, but the API container crashes in a restart loop. Source: app/routers/scopes.py, app/routers/rbac.py
• Role vocabulary drift. Client-side defaults must match the enum used by permission_engine. Hardcoded strings are fragile, as illustrated by the v0.9.0 → v0.9.1 member-add fix. Source: app/services/permissions.py
• Cross-department membership. A single user may hold several ScopeMembership rows; effective access is the union. Administrators should manage these rows through the scopes router rather than by editing memberships directly. Source: app/routers/scopes.py

Together, the three subsystems give Arkon a clean separation between *who is allowed in* (scopes), *what they can do* (RBAC), and *what they actually did* (audit).

Related Pages

Related topics: Department Scopes, RBAC & Audit, AI Skills: Distribution & Contribution Workflow

MCP Server & AI Provider Catalog

Overview

Arkon exposes its document corpus, wiki, and scope-based permission system to external AI assistants through a Model Context Protocol (MCP) server. The app/mcp/ package is the canonical home for that surface, and it is fronted by an OAuth 2.x authorization flow registered through app/routers/oauth.py. Together, these components form what this page calls the MCP Server & AI Provider Catalog — the integration boundary that lets a host (Claude Desktop, Gemini, custom agents, or self-hosted LLMs) speak to Arkon as a tool-and-resource provider rather than a generic REST API.

The catalog abstraction unifies three concerns:

1. Transport — JSON-RPC over HTTP/SSE delivered by app/mcp/server.py.
2. Capability registration — tools and resources advertised to the client, declared in app/mcp/tools.py and app/mcp/resources.py.
3. Trust & authorization — OAuth login, scope checks, and request shaping handled in app/routers/oauth.py, app/mcp/permissions.py, and app/mcp/middleware.py.

Server, Tools, and Resources

app/mcp/server.py wires the MCP application instance, mounts the tools and resources routers, and registers request/response middleware. It is the only entry point a host needs to discover; everything else is delegated to sub-modules.

app/mcp/tools.py declares the action surface Arkon exposes to the model. Tools are typically a one-to-one mapping with existing internal services (search documents, read a wiki page, list workspaces, ask a question against the corpus) and are intentionally narrow so the model can compose them.

app/mcp/resources.py declares readable resources identified by URIs (for example, arkon://wiki/{page_id} or arkon://source/{source_id}). Resources are preferred over tools when the host only needs to fetch a known artifact rather than perform an action; they are also cheaper to authorize because their access scope is static per URI scheme.

Provider Catalog and Auth Flow

The "AI Provider Catalog" half of the feature is what allows multiple LLM vendors — and increasingly self-hosted models — to authenticate against the same Arkon MCP endpoint. The OAuth router in app/routers/oauth.py is the integration point: it issues access tokens bound to a workspace, a user, and the scope set requested by the client. Tokens are then validated on every MCP request by middleware before the call reaches a tool or resource handler.

sequenceDiagram
    participant Host as MCP Host (Claude/Gemini/Custom LLM)
    participant OAuth as app/routers/oauth.py
    participant MW as app/mcp/middleware.py
    participant Server as app/mcp/server.py
    participant Tools as app/mcp/tools.py
    participant Perms as app/mcp/permissions.py
    Host->>OAuth: authorize + token request
    OAuth-->>Host: access token (scoped)
    Host->>MW: JSON-RPC call (Bearer token)
    MW->>Perms: validate scopes for tool/resource
    Perms-->>MW: allow | deny
    MW->>Server: dispatch to tool/resource
    Server->>Tools: invoke handler
    Tools-->>Host: result

The catalog is intentionally pluggable: adding a new provider means registering its OAuth client in app/routers/oauth.py and ensuring the token's audience claim matches the MCP server. No changes to server.py or tools.py are required for a new provider, which is the property the community has been asking for when requesting self-hosted LLM support.

Permissions and Middleware

app/mcp/permissions.py is the single source of truth for what an authenticated principal may do. It maps MCP method names and resource URI schemes onto the workspace-scope model that already governs the web UI, so an MCP token cannot exceed the rights of the user who authorized it. This is the mechanism that addresses community concerns raised in issues like #11 (cross-department access control): an MCP session inherits the same department/workspace boundaries visible in the UI.

app/mcp/middleware.py wraps every JSON-RPC call to provide:

• Structured logging of tool invocations and resource reads for audit.
• Rate limiting keyed by token subject, not IP, so a single user cannot bypass limits by reconnecting.
• Error normalization so internal exceptions are surfaced as MCP-compliant error codes rather than leaking stack traces to the host LLM.

Community context worth noting here: issue #7 ("Lỗi không kết nối đc claude") shows users wiring Claude Desktop to the MCP endpoint via mcp-remote; issue #19 asks for self-hosted LLM compatibility, and the catalog design above is the path to fulfilling that without forking the server. Issue #22 also demonstrates third-party MCP marketplaces consuming Arkon's tools, confirming the public contract is stable.

Related Source Files

• app/mcp/server.py — MCP application and dispatcher.
• app/mcp/tools.py — Action surface exposed to hosts.
• app/mcp/resources.py — URI-addressable resources.
• app/mcp/permissions.py — Scope-based authorization.
• app/mcp/middleware.py — Logging, limits, error shaping.
• app/routers/oauth.py — OAuth 2.x flow and token issuance.

Related Pages

Related topics: Department Scopes, RBAC & Audit, MCP Server & AI Provider Catalog

AI Skills: Distribution & Contribution Workflow

Arkon ships AI Skills as versioned, file-backed capability bundles that an LLM agent (Claude, Gemini, or a compatible runtime) can load on demand to perform specialized tasks such as editing documents, generating wikis, or querying external tools. The distribution layer governs *who* can fetch a skill and *how* it is exposed to agents, while the contribution layer governs *how* new or updated skills move from an internal author into a deployable artifact. Source: docs/SKILLS.md:1-40.

Skill Packaging and On-Disk Layout

A Skill is a directory whose root contains a manifest named SKILL.md. The manifest declares the skill's identity, a natural-language description (used by the agent for retrieval), and the list of resources (prompts, scripts, schemas) that the agent may invoke. Source: skills/arkon-edit/SKILL.md:1-15.

The repository follows a convention where each skill lives under skills/<skill-name>/SKILL.md, which mirrors what the runtime expects when mounting a skill catalog. The bundled arkon-edit skill provides the canonical example of this layout and serves as the reference implementation for downstream contributors. Source: skills/arkon-edit/SKILL.md:1-15.

Seed scripts populate the catalog from these on-disk bundles during installation or upgrade. seed_skills.py walks the skills/ directory, parses each SKILL.md, and inserts or updates the corresponding database record, ensuring that the runtime catalog matches the repository state. Source: app/scripts/seed_skills.py:1-80.

Distribution Layer

Distribution is implemented as a FastAPI router that exposes skill metadata and content over HTTP, gated by authentication and (where configured) scope-based authorization. The router delegates all persistence and validation to SkillService, which encapsulates CRUD, versioning, and rendering. Source: app/routers/skills.py:1-60.

SkillService is the single source of truth for skill state: it tracks the current published version, draft versions pending review, and the per-scope visibility list that determines which workspaces or departments can read the skill. Reading a skill through the service returns both the manifest and any inline resources, so an MCP client receives everything it needs in one round trip. Source: app/services/skill_service.py:1-120.

flowchart LR
  Agent[LLM Agent / MCP Client] -->|GET /api/skills| Router[skills router]
  Router --> Service[SkillService]
  Service --> DB[(PostgreSQL)]
  Service --> FS[(skills/ bundle)]
  Seed[seed_skills.py] --> DB
  Contrib[skill_contributions router] --> Service

The diagram above shows the request path for a skill fetch and the two write paths (seeding from disk, contributions from API). Source: app/routers/skills.py:1-60, app/services/skill_service.py:1-120, app/scripts/seed_skills.py:1-80.

Contribution Workflow

Internal authors do not commit directly to the published catalog. Instead, they submit a contribution through skill_contributions.py, which models the proposal as a discrete record with fields such as skill_name, target_version, manifest_diff, author_id, and status. Source: app/routers/skill_contributions.py:1-80.

The workflow progresses through states:

1. Draft – the author composes a new SKILL.md and uploads it via POST /api/skill-contributions. The payload is validated against the manifest schema and stored as a draft.
2. Review – a reviewer (typically a workspace admin) fetches the draft, inspects the manifest and resources, and either requests changes or approves.
3. Published – on approval, SkillService materializes the draft into the live catalog, bumping the version and invalidating downstream caches.

Source: app/routers/skill_contributions.py:80-160.

Each transition is recorded so that audits can reconstruct who changed what and when. The contribution router delegates persistence to SkillService so that validation rules (manifest shape, required fields, resource references) stay consistent between seed-time imports and runtime submissions. Source: app/services/skill_service.py:120-200.

Operational Concerns

Community feedback highlights two recurring operational concerns that the distribution layer is designed to address:

• Skill leakage across tenants. Issue #5 raised the risk that proprietary skills trained on internal data could be exfiltrated when shared with contractors. The skill router scopes every read by the caller's workspace membership, and SkillService refuses to serialize a skill whose scope list does not include the requester. Source: app/routers/skills.py:60-120, app/services/skill_service.py:200-260.
• Compatibility with the Claude MCP runtime. Issue #7 reports MCP handshake failures when the local Claude Desktop cannot reach the Arkon skill endpoint. The router returns the manifest in the exact shape that Claude's MCP client expects, including the resource block descriptors that the runtime uses to mount tools. Source: app/routers/skills.py:120-180.

Seeding is idempotent: re-running seed_skills.py updates existing skills in place and never duplicates records, which makes it safe to invoke from CI on every deploy. Source: app/scripts/seed_skills.py:80-160.

Summary

The Skills subsystem separates *packaging* (the SKILL.md on-disk format), *distribution* (skills router plus SkillService), and *contribution* (skill_contributions router plus the draft/review/published state machine). Authors iterate through the contribution workflow, reviewers gate publication, and the seeded catalog plus the API together present a single, scoped view of available skills to LLM agents. Source: docs/SKILLS.md:40-80, app/services/skill_service.py:260-320, app/routers/skill_contributions.py:160-240.

Related Pages

Related topics: System Architecture & Tech Stack, MRP Pipeline: Document Ingestion & Wiki Compilation, Wiki Browser, Drafts & Branches, Department Scopes, RBAC & Audit

Deployment, Configuration & Troubleshooting

Overview

Arkon ships as a containerized stack composed of a backend API, a frontend, and a PostgreSQL database. The deployment surface is intentionally small: a multi-stage Dockerfile, a docker-compose.yml orchestrator, and .env-driven configuration. This page documents how to deploy, configure, and diagnose the most common failure modes observed in the community, including login issues, missing routers, and document ingestion errors.

Deployment

Docker Compose Stack

The recommended path is a single command: docker compose --env-file .env.docker up -d --build. The compose file wires three services together — db (PostgreSQL), backend (FastAPI), and frontend (web UI) — and exposes ports for local access. Source: docker-compose.yml:1-120

Backend Image Build

The Dockerfile is multi-stage: a builder stage installs Python dependencies declared in pyproject.toml, and the runtime stage copies the prepared virtualenv, applies Alembic migrations on container start, and launches Uvicorn. The startup script entrypoint.sh is the canonical entrypoint and is responsible for running alembic upgrade head before the API binds. Source: Dockerfile:1-80, Source: entrypoint.sh:1-40

Local Development

For non-containerized work, .env.local.example provides the variable template consumed by Uvicorn directly. Developers typically run:

cp .env.local.example .env
alembic upgrade head
uvicorn app.main:app --reload

Source: docs/SETUP.md:1-60, Source: alembic.ini:1-40

Configuration

Environment Variables

Configuration is entirely environment-driven. The two reference files split variables into runtime domains:

Arkon v0.9.0 removed the legacy "Project" layer; the remaining authorization model is Workspaces + Scopes (departments), which depends on the scopes router being active. Source: .env.docker.example:1-80, Source: .env.local.example:1-60

Database & Migrations

DATABASE_URL must use the postgresql+asyncpg driver because the application uses async SQLAlchemy. Migrations live under alembic/versions and are applied automatically by entrypoint.sh before the server starts. A failing migration aborts the container boot loop. Source: alembic.ini:1-40, Source: entrypoint.sh:1-40

Troubleshooting

Login Failures After Fresh Deploy

The most reported deployment failure is "wrong password" on first login even with the supplied .env. The cause is almost always a mismatch between POSTGRES_PASSWORD (used by the database service) and the password embedded in DATABASE_URL (used by the backend). Both must agree, and the URL must URL-encode special characters. Source: docker-compose.yml:1-120, Source: .env.docker.example:1-80

API 404 on `/api/scopes/*`

Issue #8 reported that adding Access Control Departments silently failed because the scopes router was never imported in app/main.py. Issue #10 extended this: after the router was registered, a fresh clone crashed the API because app/api/scopes.py referenced Action, ScopeMembership, and ScopeRole names that did not exist in app/database/models.py. The fix required exporting those symbols from the models module. Source: app/main.py:1-80, Source: app/api/scopes.py:1-60, Source: app/database/models.py:1-80

Wiki Page Creation Returns 500

Issue #20 documents POST /api/wiki/pages failing with a 500 caused by a page_type SQLAlchemy hybrid_property lacking a setter. When the route attempted to assign to page_type, SQLAlchemy raised AttributeError. Verify that the migration introducing the hybrid property includes a corresponding @page_type.setter. Source: app/api/wiki.py:1-80, Source: app/database/models.py:1-80

Document Indexing Crashes on NUL Bytes

Issue #21 shows ingestion failing with CharacterNotInRepertoireError because PostgreSQL text/varchar and JSON columns reject \x00. The mitigation is to strip NUL bytes from Source.full_text before persistence, typically in the extraction pipeline. Source: app/database/models.py:1-80

Cannot Connect to Claude / Gemini

Issue #7 reports the UI surfacing a connection error when invoking LLM-backed features. The API keys (ANTHROPIC_API_KEY, GEMINI_API_KEY) must be present in the backend container's environment and reachable from the host. After editing .env.docker, rebuild with --build so the new variables reach the container. Source: .env.docker.example:1-80

Diagnostic Workflow

flowchart TD
    A[Container fails to start] --> B{Check logs}
    B --> C[entrypoint.sh: alembic error?]
    C -- yes --> D[Fix migration / DATABASE_URL]
    C -- no --> E[Uvicorn import error?]
    E -- yes --> F[Missing symbol in app/api or app/database/models]
    E -- no --> G[Port / network conflict]
    A2[Login fails] --> H{POSTGRES_PASSWORD == DATABASE_URL password?}
    H -- no --> I[Align env values, rebuild]
    H -- yes --> J[Inspect auth route logs]
    A3[Feature returns 500] --> K{Stack trace points to hybrid_property or model field?}
    K -- yes --> L[Add missing @setter or strip NUL bytes]
    K -- no --> M[Check router registration in app/main.py]

Source: entrypoint.sh:1-40, Source: app/main.py:1-80, Source: .env.docker.example:1-80

Operational Checklist

• docker compose --env-file .env.docker up -d --build after any .env change.
• Confirm both PostgreSQL and backend agree on credentials and host.
• After router changes, restart with --build and verify endpoints via /docs.
• When adding models, export every name imported by routers (Action, ScopeMembership, ScopeRole).
• When ingesting documents, sanitize extracted text for \x00 before persisting.
• For wiki pages using hybrid properties, ensure both getter and setter are defined.

Source: docker-compose.yml:1-120, Source: pyproject.toml:1-80

Doramagic Pitfall Log

Found 13 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/20

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/21

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/nduckmink/arkon

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/nduckmink/arkon

5. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/nduckmink/arkon

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/nduckmink/arkon

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/nduckmink/arkon

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/10

9. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/7

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/22

11. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/nduckmink/arkon/issues/6

12. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/nduckmink/arkon

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using arkon with real data or production workflows.

• MarketNow — MCP skill marketplace with trust layer (8,560 skills, Sentin - github / github_issue
• Document indexing fails on NUL byte (0x00) in extracted text — Character - github / github_issue
• Creating a wiki page fails with 500 — page_type hybrid_property has no s - github / github_issue
• Lỗi edit document - github / github_issue
• [[Question]Phân quyền người dùng liên phòng ban](https://github.com/nduckmink/arkon/issues/11) - github / github_issue
• Wiki - github / github_issue
• API crash loop on fresh clone: scopes router imports undefined names (Ac - github / github_issue
• Lỗi không add Access Control Departments - github / github_issue
• Lỗi không kết nối đc claude - github / github_issue
• security issue - github / github_issue
• không thể login được - github / github_issue
• Lỗi - github / github_issue