Related Pages

Related topics: Search Engines and Indexing Pipeline, MCP Server, CLI, and Agent Integration, Authentication, Authorization, and Deployment

System Architecture and Code Layout

Purpose and Scope

codebeam is organized as a single-binary Go application whose top-level goal is to provide a code-oriented indexing and serving capability exposed through a local web UI. The repository uses the canonical Go project layout: a thin cmd/ entry point that delegates to internal packages under internal/, with static assets co-located at the module root and bundled into the binary.

The architecture separates concerns into four orthogonal layers: command bootstrapping, configuration loading, data indexing, and HTTP serving. Each layer has a dedicated package and a single public surface, which keeps the dependency graph acyclic and makes the binary straightforward to read from top to bottom.

As of the v0.2.0 release, this layout is the canonical structure referenced by the project's documentation and community guidance.

Entry Point and Process Bootstrap

The process is launched through cmd/codebeam/main.go, which is the only main package in the module. Its responsibilities are limited to: parsing flags, loading configuration, initializing the indexer, mounting the web server, and handling process-level signals for graceful shutdown.

Source: cmd/codebeam/main.go:1-80

By keeping main.go thin and pushing all real logic into internal/, the project follows the same convention used by tools such as Hugo and Cobra-based CLIs. This makes the command easy to test and to replace (for example, to embed codebeam as a library in another Go program) without dragging UI concerns into the caller.

Internal Package Layout

The internal/ tree holds four packages that together implement every feature of the tool:

• internal/config — Defines the configuration struct, default values, and flag bindings. The config package is the foundation everything else depends on.
• internal/store — Provides the persistence layer (typically backed by a BoltDB or SQLite-style local store) where indexed files, metadata, and search-related records live between requests.
• internal/indexer — Walks one or more source directories, extracts symbols or text chunks, and writes them into the store. This is the ingest path.
• internal/web — Hosts the HTTP server, route definitions, request handlers, and renders the embedded UI bundled from embed.go.

The dependency direction is strictly top-down: cmd → web → store ↔ indexer, with config consumed by the higher layers. There are no cycles among these packages.

Source: internal/config/config.go:1-60 Source: internal/store/store.go:1-80 Source: internal/indexer/indexer.go:1-100

Embedded Assets and Distribution

Static frontend assets (HTML, CSS, JavaScript, and any icons or templates) are not vendored as separate files in the released binary. Instead, the repository declares an embed.FS at the module root via embed.go, which is consumed by the web layer. This produces a single self-contained codebeam binary with no external asset directory required at runtime — a property that is repeatedly cited in community discussions as a quality-of-life improvement over v0.1.x.

Source: embed.go:1-30

The web package retrieves the embedded filesystem and serves it either as raw bytes for API endpoints or as a http.FileServer rooted at the same FS for the SPA shell.

Component Interaction Flow

The end-to-end flow on a typical run is: the operator launches the binary, which loads config, opens the store, performs an initial index pass over the configured path, and then starts the HTTP server. Subsequent re-index runs are triggered either by a CLI flag or by a route handler.

flowchart TD
    A[main.go] --> B[config.Load]
    B --> C[store.Open]
    C --> D[indexer.Index]
    D --> C
    A --> E[web.Server]
    E --> C
    E --> F[embed.FS]
    E --> G[HTTP Client]

Source: cmd/codebeam/main.go:40-120 Source: internal/web/server.go:1-80

Layout Summary

The repository's structure communicates its architecture directly through its package names. A reader can infer the system's behavior from the directory tree alone: cmd is the entry, internal/config is the policy, internal/store is the memory, internal/indexer is the ingest engine, internal/web is the interface, and embed.go provides the visuals. This explicit, flat layout is part of why the project remains approachable to new contributors and is referenced in onboarding discussions for the v0.2.0 release.

Related Pages

Related topics: System Architecture and Code Layout, MCP Server, CLI, and Agent Integration

Search Engines and Indexing Pipeline

The Search Engines and Indexing Pipeline is the core data-processing backbone of codebeam. Its job is to turn raw source code in a repository into a searchable, structurally aware index, and then to resolve user queries against that index with consistent normalization. The pipeline is split across three internal packages — structural, indexer, and search — each responsible for one stage of the flow.

Pipeline Overview

The pipeline runs in two phases: an indexing phase (one-shot or incremental) and a query phase (interactive). The structural engine parses source files, the indexer packages the parsed output into blobs, and the search package normalizes both indexed content and incoming queries so that matches are comparable.

flowchart LR
    A[Source Files] --> B[structural/engine.go<br/>structural/wasm.go]
    B --> C[Parse Trees + Symbols]
    C --> D[indexer/indexer.go]
    D --> E[indexer/blobs.go<br/>Blob Records]
    E --> F[Search Index]
    G[User Query] --> H[search/normalize.go]
    H --> I[search/search.go]
    I --> F
    F --> J[Ranked Results]

Structural Engine

The structural engine is the front end of the pipeline. It consumes source files and produces parse trees and symbol information that downstream stages can index. The presence of internal/structural/wasm.go indicates that parsing is delegated to a WebAssembly runtime, which is a common way to embed language parsers (such as tree-sitter grammars) without native dependencies. Source: internal/structural/engine.go, internal/structural/wasm.go.

Responsibilities:

• Dispatch a file to the appropriate language grammar based on extension or content sniffing.
• Drive the WASM parser instance and convert its output into a normalized in-memory representation.
• Emit structural units (functions, classes, modules, blocks) that the indexer can record as discrete blobs.

The engine is the only stage that is language-aware; everything after it operates on the engine's neutral representation. Source: internal/structural/engine.go.

Indexer and Blob Storage

The indexer takes the structural units produced by the engine and persists them as blobs. internal/indexer/indexer.go orchestrates the walk over a repository, deciding which files need re-indexing and how to batch work, while internal/indexer/blobs.go defines the blob record format. Source: internal/indexer/indexer.go, internal/indexer/blobs.go.

A blob is the atomic searchable unit — typically a function, class, or top-level declaration — paired with the metadata needed to surface it (file path, language, byte ranges, symbol name). Splitting storage this way keeps query latency low because each blob can be scored and returned independently.

Key behaviors supported by the indexer package:

• Incremental re-indexing driven by file change detection.
• Stable identity for blobs so that updates replace rather than duplicate records.
• A blob schema that downstream search can iterate without re-parsing.

Search and Normalization

The search package resolves user queries against the indexed blobs. internal/search/normalize.go is applied symmetrically to both indexed content and incoming queries, which guarantees that matching is symmetric — a token that survives normalization on the way in also survives it on the way out. Source: internal/search/normalize.go, internal/search/search.go.

Normalization typically covers case folding, identifier splitting (so getUserName matches get user name or getusername), punctuation stripping, and whitespace collapsing. The exact rules live in normalize.go; the matcher in search.go consumes the normalized streams and ranks results. Source: internal/search/search.go.

The search stage is intentionally stateless beyond the index it reads, which keeps the package easy to test and reuse from CLI, MCP, or HTTP entrypoints.

Release Context (v0.2.0)

The v0.2.0 release (compared against v0.1.1) is the current published version of the pipeline described above. When evaluating behavior or upgrading from an earlier version, the diff at v0.1.1...v0.2.0 is the authoritative reference for what changed in the structural, indexer, and search packages.

Practical Notes for Contributors

• Add new languages by extending the structural engine; the indexer and search layers should not need changes because they consume a neutral shape. Source: internal/structural/engine.go.
• Changes to the blob schema in internal/indexer/blobs.go are breaking and require an index migration or full rebuild. Source: internal/indexer/blobs.go.
• Any normalization rule added to internal/search/normalize.go must be applied to both indexed blobs and queries, or match symmetry will break. Source: internal/search/normalize.go.
• Keep WASM parser glue isolated in internal/structural/wasm.go so the engine can be unit-tested without the runtime. Source: internal/structural/wasm.go.

This separation — parser, storage, query — is what allows codebeam to keep the indexing path fast and the query path predictable while remaining extensible to new languages and frontends.

Related Pages

Related topics: System Architecture and Code Layout, Search Engines and Indexing Pipeline, Authentication, Authorization, and Deployment

MCP Server, CLI, and Agent Integration

Overview and Scope

The codebeam project ships three coordinated entry points that together implement the Model Context Protocol (MCP) surface and a paired command-line interface: a long-running MCP server embedded in the main binary, an HTTP/web transport wrapper for the same protocol, and a lightweight cb agent binary that consumes MCP context. The CLI subcommands in cmd/codebeam (mcp, ctx) and the dispatch logic in internal/cli/cli.go orchestrate which surface is exposed. As of release v0.2.0 this integration is the primary external interface, allowing external agents and editors to query project structure, symbols, and context through a standardized protocol. Source: cmd/codebeam/mcp.go:1-40, cmd/codebeam/ctx.go:1-40, internal/cli/cli.go:1-60.

MCP Server Implementation

The core MCP server lives in the internal mcp package and is the protocol-correct entry point used by both the CLI's mcp subcommand and the web transport layer. It registers the request handlers that respond to MCP initialize, tools/list, resources/read, and related methods, and it is intentionally transport-agnostic so that the same server can be reused by the stdio and HTTP front-ends. Source: internal/mcp/mcp.go:1-80.

The web transport in internal/web/mcp.go mounts the same server behind an HTTP endpoint, exposing the MCP JSON-RPC surface to browser-based clients and remote agents. This split keeps the protocol logic in one place while letting users pick stdio (for local editor integration) or HTTP (for remote/headless usage). Source: internal/web/mcp.go:1-60.

The CLI invocation codebeam mcp simply wires the MCP server onto stdio (or the configured transport) and blocks until the client disconnects; there is no HTTP listener when launched through this path. Source: cmd/codebeam/mcp.go:1-40.

CLI Surface and the `ctx` Command

internal/cli/cli.go is the dispatch root. It parses the top-level subcommand, configures logging, and delegates to the handlers in cmd/codebeam. The ctx subcommand in cmd/codebeam/ctx.go is the user-facing companion to the MCP server: it produces a snapshot of the project's contextual information (files, symbols, configuration) that the MCP server later serves to agents. Separating ctx (build context) from mcp (serve context) means expensive context generation is decoupled from the long-lived server process and can be cached. Source: cmd/codebeam/ctx.go:1-60, internal/cli/cli.go:1-80.

A typical workflow is:

1. User runs codebeam ctx to build/refresh the context index.
2. User runs codebeam mcp (or starts the web transport) to serve that context.
3. An external agent connects over MCP and calls resources/tools.

This sequence lets a developer iterate on the context snapshot without restarting the server, and lets the server be started independently in production. Source: cmd/codebeam/ctx.go:20-60, cmd/codebeam/mcp.go:1-40.

The `cb` Agent Binary

cmd/cb/main.go is a separate, single-purpose binary whose role is complementary to the main codebeam CLI: rather than serving context, cb consumes it. It embeds a minimal MCP client and is intended for use inside agent loops, CI jobs, or scripted automation where a small, dependency-light tool is preferable to spawning the full CLI. By keeping cb as its own main, the project avoids pulling CLI-only dependencies (such as the full argument parser in internal/cli/cli.go) into the agent hot path. Source: cmd/cb/main.go:1-60, internal/cli/cli.go:1-60.

The two binaries share no process state; they communicate exclusively through the MCP protocol over a transport chosen at launch. This loose coupling is the central design property of the integration: codebeam produces and serves, cb consumes and acts, and either side can be replaced as long as it speaks MCP.

Architecture Summary

flowchart LR
  A[codebeam ctx<br/>build context] --> B[(Context<br/>snapshot)]
  B --> C[codebeam mcp<br/>MCP server<br/>internal/mcp/mcp.go]
  C -->|stdio| D[Editor / Agent]
  C -->|HTTP| E[internal/web/mcp.go]
  E --> F[Remote Agent]
  G[cb agent<br/>cmd/cb/main.go] -->|MCP client| C
  G -->|MCP client| E
  D --> H[internal/cli/cli.go dispatcher]
  F --> H

The dispatcher in internal/cli/cli.go is the only piece shared across all three entry points, which is why the mcp and ctx handlers are kept thin and delegate their work to the internal/mcp and context packages respectively. Source: internal/cli/cli.go:1-80, cmd/codebeam/mcp.go:1-40, cmd/codebeam/ctx.go:1-60, internal/mcp/mcp.go:1-80, internal/web/mcp.go:1-60, cmd/cb/main.go:1-60.

Versioning Notes

The integration described above was stabilized in v0.2.0. The changelog between v0.1.1 and v0.2.0 is the canonical reference for protocol additions, transport changes, and any breakage of the ctx snapshot format; consumers such as cb should be kept on a compatible protocol version when upgrading. Source: release notes v0.2.0.

Related Pages

Related topics: System Architecture and Code Layout, MCP Server, CLI, and Agent Integration

Authentication, Authorization, and Deployment

Codebeam exposes two distinct authentication surfaces — a CLI-oriented OAuth device/authorization-code helper and a browser-based OIDC login flow for the embedded web UI — and persists authorization state through a dedicated store. This page documents how those pieces fit together and what operators must configure to deploy them.

1. Subsystem Map

The authentication code is split along the same seam as the rest of the project: CLI logic lives under internal/cli/, HTTP handlers under internal/web/, and persistence under internal/store/. The OIDC primitive used by both is factored into internal/oidc/ so neither surface re-implements discovery, token exchange, or ID-token verification.

Source: internal/oidc/oidc.go for the shared primitive; Source: internal/cli/oauth.go and Source: internal/web/oauth.go for the two surfaces.

2. CLI Authentication (`codebeam login`)

The CLI surface in internal/cli/oauth.go is responsible for turning a fresh install into an authenticated client. It does not embed its own user interface; instead it orchestrates the standard OAuth 2.0 authorization-code flow against an external identity provider configured at deployment time.

Typical responsibilities visible from the file's structure:

• Parsing the OIDC issuer / client ID / client secret / redirect configuration supplied via flags or environment variables.
• Performing discovery against ${ISSUER}/.well-known/openid-configuration using the helper in internal/oidc/oidc.go.
• Spinning up a short-lived local HTTP listener to receive the authorization code, then exchanging it for tokens.
• Handing the resulting token bundle to internal/store/auth.go for durable persistence so subsequent CLI invocations can re-authenticate silently (for example via refresh tokens) without re-prompting the user.

Source: internal/cli/oauth.go and Source: internal/oidc/oidc.go. Persistence behavior: Source: internal/store/auth.go.

3. Web Authentication (OIDC Login)

The web surface mirrors the CLI flow but is initiated from a browser. internal/web/oauth.go and internal/web/oidc.go together implement the routes, while internal/web/oidc_login_test.go exercises them end-to-end.

The expected workflow:

sequenceDiagram
    participant Browser
    participant Web as internal/web (oidc.go)
    participant Store as internal/store (auth.go)
    Browser->>Web: GET /login
    Web->>Browser: 302 redirect to IdP (authorization endpoint)
    Browser->>IdP: authenticate
    IdP->>Browser: redirect with ?code=
    Browser->>Web: GET /callback?code=...
    Web->>Store: persist session / tokens
    Web->>Browser: set session cookie, redirect to app

Key points supported by the source layout:

• The login route is intentionally thin: it delegates discovery, PKCE, and token verification to internal/oidc/oidc.go rather than re-implementing them per handler.
• Authorization on subsequent requests is enforced by middleware that reads the session established in auth.go, not by re-running the OIDC dance on every request.
• The presence of a dedicated oidc_login_test.go indicates the callback handler's state-machine behavior (success, error, missing code, expired state) is treated as a contract.

Source: internal/web/oidc.go, Source: internal/web/oauth.go, Source: internal/web/oidc_login_test.go, Source: internal/store/auth.go.

4. Authorization Model and Deployment

The store layer in internal/store/auth.go is the single source of truth for "who is the caller" once authentication has completed. Both the CLI and the web layer write into it, and both read from it when authorizing subsequent operations.

Deployment implications drawn from the layout:

• Identity provider: A single OIDC issuer is configured for the deployment and consumed by both surfaces; misconfiguration in internal/oidc/oidc.go affects CLI and web symmetrically.
• Secrets: Client secrets for the CLI's public-client flow and the web's confidential-client flow must be supplied out-of-band; the store layer assumes tokens, not raw credentials, are what it persists.
• Session lifecycle: Because auth.go owns persistence, deployers must ensure the backing store (file, database, or KV) survives restarts and is reachable from both the CLI process and the web process in shared-host deployments.
• Upgrade note (v0.2.0): The latest release (v0.1.1...v0.2.0) sits on top of this auth stack; operators upgrading should re-run codebeam login so refresh tokens written by the previous schema are replaced by v0.2.0-compatible entries in the store.

Source: internal/store/auth.go, Source: internal/oidc/oidc.go, Source: internal/cli/oauth.go. Upgrade context: Source: CHANGELOG / release v0.2.0.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: identity.distribution | https://github.com/clement-tourriere/codebeam

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/clement-tourriere/codebeam

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/clement-tourriere/codebeam

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/clement-tourriere/codebeam

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/clement-tourriere/codebeam

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/clement-tourriere/codebeam

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/clement-tourriere/codebeam

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using codebeam with real data or production workflows.

• v0.2.0 - github / github_release
• v0.1.1 - github / github_release
• v0.1.0 - github / github_release
• Installation risk requires verification - GitHub / issue