Related Pages

Related topics: Core Architecture: Retrieval, Fusion & Cognitive Layer, Deployment, Security, Operations & Extensibility

Project Overview & Installation

DepthFusion is a hybrid retrieval framework that fuses classical lexical scoring (BM25) with approximate nearest-neighbour (ANN) vector search over dense embeddings. The project ships a Python library, a long-running REST API service suitable for production, and a CLI generated from the same surface area. The current major release is DepthFusion v2.0.0, while v1.2.0 introduced the foundational HNSW embedding index, CI hardening, security updates, and the systemd-deployable REST service. Source: README.md:1-40.

Purpose and Scope

DepthFusion targets use cases where inverted-index recall and embedding-based semantic recall must be combined into a single ranking. Its core contract (referenced in the v1.2.0 release notes as the *ruflo-mod contract*) defines a fused BM25-HNSW recall pipeline: a query is scored by both a sparse lexical retriever and a dense HNSWStore index, and the two score streams are merged before results are returned. Source: README.md:42-78.

The system is organised around three primary surfaces:

• Library API — importable Python modules that expose index construction, ingestion, and fused querying.
• REST API service — an HTTP server backed by a systemd unit, added in v1.2.0 to support always-on deployments.
• CLI — a generated command-line interface that mirrors the library entry points, so the same operations available in code can be invoked from shell scripts.

The repository additionally publishes install scripts tuned for two environments: a developer workstation and a virtual private server (VPS). Source: INSTALL.md:1-30.

System Requirements

DepthFusion's Python components require CPython 3.10 or newer, since the codebase relies on modern type-hint syntax and pattern matching. The HNSW index is built on top of hnswlib, which links against native BLAS/OpenMP runtimes; on Linux distributions this typically means libgomp1 must be present before the Python wheel can be imported. Source: INSTALLATION.md:12-26.

For the REST service, an OS with systemd (any modern Debian-, Ubuntu-, RHEL-, or Arch-derived distribution) is recommended so that the bundled unit file can manage lifecycle, logging, and restart behaviour. Source: INSTALLATION.md:28-44.

Hardware expectations scale with embedding dimensionality and corpus size. A workstation with 16 GB of RAM is sufficient for small-to-medium indices used during development, while VPS deployments of production corpora should provision at least 4 vCPU and 8 GB of RAM to keep index rebuilds and concurrent queries responsive. Source: INSTALL.md:32-58.

Installation Methods

The repository exposes three install entry points so the same project can be brought up on a developer laptop, a clean container, or a hardened VPS without duplicating manual steps.

Source: setup.sh:1-40, scripts/install.sh:1-50, scripts/install-vps.sh:1-60.

Local Developer Install

setup.sh creates a project-local .venv, upgrades pip, and installs DepthFusion in editable mode so that source edits take effect without a reinstall. It also prints the exact command needed to activate the environment. Source: setup.sh:8-34.

Clean-Host Install

scripts/install.sh is the canonical entry point for CI runners and fresh containers. It detects the package manager, installs required system libraries (notably libgomp1 for hnswlib), creates a service user, and provisions the Python environment under /opt/depthfusion. The script is safe to re-run; each step is guarded so that an existing installation is upgraded rather than duplicated. Source: scripts/install.sh:18-72.

VPS Install

scripts/install-vps.sh extends the clean-host flow with production hardening: it registers a systemd unit so the REST API starts on boot, opens only the configured HTTP port through the host firewall, and writes a default configuration file under /etc/depthfusion/config.toml. Running this script is the documented path for deploying the v1.2.0 REST API service. Source: scripts/install-vps.sh:22-88.

Post-Installation Steps

After the chosen script finishes, three follow-up actions are required before the service can serve real traffic:

1. Populate the index — ingest documents using the CLI or library API. Dense vectors are written to an HNSWStore directory while the lexical index is materialised alongside it.
2. Edit the configuration — review /etc/depthfusion/config.toml (or the equivalent path on non-VPS installs) to set index locations, port bindings, and fusion weights.
3. Start the service — on systemd hosts, run systemctl enable --now depthfusion; otherwise, launch the server module directly from the activated virtual environment.

Source: INSTALL.md:60-96, INSTALLATION.md:46-78.

Operational Notes

Because v1.2.0 introduced both the HNSWStore and the systemd-backed REST service together, operators upgrading from earlier minor versions should treat the index directory as a load-bearing artifact and back it up before restarting the service. The release notes flag this transition explicitly so that the dense index and the lexical index stay in sync after a re-deploy. Source: INSTALLATION.md:80-104.

Related Pages

Related topics: Tauri Desktop App & 30-Tool MCP Surface, Deployment, Security, Operations & Extensibility

Core Architecture: Retrieval, Fusion & Cognitive Layer

The Retrieval, Fusion & Cognitive Layer is the central nervous system of DepthFusion's answer-generation pipeline. It is responsible for turning a natural-language query into a ranked, security-filtered set of context chunks that downstream reasoning components consume. The architecture is organised into three cooperating sub-layers: a Retrieval Layer that produces candidate sets using complementary sparse and dense signals, a Fusion Layer that merges those signals into a single ranked list, and a Cognitive Layer that reranks candidates and enforces access-control policy before any chunk reaches a generator. The design follows the BM25-HNSW fused recall contract introduced in the v1.2.0 release (E-45) and remains the contract surface for v2.0.0. Source: src/depthfusion/retrieval/hybrid.py:1-40

Retrieval Layer

The Retrieval Layer produces two independent candidate streams in parallel so that lexical precision and semantic recall both contribute to recall quality.

BM25 Sparse Retriever

bm25.py implements the lexical half of recall. It tokenises corpus text, builds inverted-style statistics (term frequencies, document lengths, IDF), and scores queries against the corpus using the Okapi BM25 ranking function. Because BM25 depends on exact term overlap, it excels at queries that name entities, identifiers, or rare technical vocabulary. The retriever exposes a search(query, top_k) method returning (doc_id, score) tuples ordered by descending BM25 score. Source: src/depthfusion/retrieval/bm25.py:1-80

HNSW Dense Retriever

hnsw_store.py wraps hnswlib to provide an approximate-nearest-neighbour (ANN) index over dense embeddings, fulfilling the E-45 contract. The store supports add_items, knn_query, and persistence to disk. At query time it returns the top-k nearest neighbours by cosine or inner-product similarity, depending on configuration. The HNSW backend was selected for sub-linear query latency over millions of vectors, which is essential for the v2.0.0 release's larger corpora. Source: src/depthfusion/retrieval/hnsw_store.py:1-90

Hybrid Coordinator

hybrid.py orchestrates both retrievers. It issues the same query to BM25 and to the HNSW store, gathers each top-k candidate set, and forwards them to the Fusion Layer together with the original query. The coordinator also normalises top-k values so downstream fusion operates on comparable list lengths. Source: src/depthfusion/retrieval/hybrid.py:40-120

Fusion Layer

The Fusion Layer merges the BM25 and HNSW candidate lists into a single ranked list. DepthFusion uses Reciprocal Rank Fusion (RRF) as implemented in fusion/rrf.py. RRF is parameter-light and robust to score-scale differences, which is precisely the situation when combining BM25 scores (unbounded, term-dependent) with HNSW similarity scores (bounded in [-1, 1] or non-negative).

The RRF formula aggregates per-list ranks as score(d) = Σ 1 / (k + rank_i(d)), where k is a smoothing constant (typically 60). Documents not present in a given list contribute nothing for that list. The result is a fused ranking that rewards documents appearing in both lists, satisfying the BM25-HNSW fused recall contract. Source: src/depthfusion/fusion/rrf.py:1-60

The fused list is then handed to the Cognitive Layer for reranking and policy enforcement.

Cognitive Layer

The Cognitive Layer applies two operations: a learned reranker that improves precision at the top of the list, and an ACL verifier that prunes any chunk the current principal is not authorised to read.

Reranker

reranker.py consumes the fused candidate list and rescores documents using a cross-encoder or comparable model. Cross-encoders jointly encode (query, document) pairs and produce calibrated relevance scores that typically outperform the retrieval-only ranking by a wide margin at rank 1-10. The reranker exposes rerank(query, documents, top_k) and returns the reordered subset. Source: src/depthfusion/retrieval/reranker.py:1-80

ACL Verifier

acl_verifier.py is the final gate. Each candidate carries an ACL descriptor (tenant, classification, share list) that the verifier compares against the caller's identity and policy context. Chunks that fail the check are dropped silently from the result set; the remaining chunks constitute the context window for the generator. This stage guarantees that fused recall cannot bypass authorisation, even if a high-scoring chunk originates from a restricted partition. Source: src/depthfusion/retrieval/acl_verifier.py:1-70

End-to-End Data Flow

The diagram below traces a single query from arrival to context emission.

flowchart LR
    Q[Query] --> H[hybrid.py coordinator]
    H --> B[bm25.py sparse retriever]
    H --> N[hnsw_store.py dense retriever]
    B --> F[fusion/rrf.py RRF merge]
    N --> F
    F --> R[reranker.py cross-encoder]
    R --> A[acl_verifier.py policy gate]
    A --> C[Context chunks for generator]

The pipeline is deliberately staged so that each layer is independently testable and replaceable. Swapping the BM25 backend, changing the fusion strategy from RRF to a weighted combination, or substituting the reranker does not require touching the ACL verifier, which keeps the security boundary stable across releases.

Configuration & Contract Surface

The interface between layers is narrow: (query, top_k) -> [(doc_id, score)]. Each component honours the contract from v1.2.0's E-45 BM25-HNSW fused recall specification, which v2.0.0 inherits unchanged. Operators tune recall quality primarily through three knobs: the BM25 (k1, b) parameters, the HNSW (ef_construction, ef_search, M) parameters, and the reranker's top_k. The RRF k constant and ACL policy descriptors are the remaining levers. Because the contract is stable, the REST API service introduced in v1.2.0 and the generated CLI in v2.0.0 both invoke the same pipeline without separate code paths. Source: src/depthfusion/retrieval/hybrid.py:120-160

Summary

DepthFusion's Retrieval, Fusion & Cognitive Layer delivers a three-stage pipeline: parallel sparse/dense retrieval, RRF-based fusion, and reranking followed by ACL enforcement. Each stage is isolated in its own module under src/depthfusion/retrieval/ and src/depthfusion/fusion/, which keeps the contract surface stable from E-45 through the v2.0.0 release and supports the REST API and CLI front-ends introduced in v1.2.0 and v2.0.0 respectively.

Related Pages

Related topics: Core Architecture: Retrieval, Fusion & Cognitive Layer, Deployment, Security, Operations & Extensibility

Tauri Desktop App & 30-Tool MCP Surface

DepthFusion ships a single coherent product surface that pairs a Tauri-based desktop client with a Model Context Protocol (MCP) server exposing a curated catalog of 30 tools. The two halves share one underlying Python core, so any operation available through the graphical interface is also reachable verbatim by an external agent speaking the MCP wire protocol. This page documents both surfaces and how they interlock.

1. Tauri Shell and Frontend Composition

The desktop client is a Tauri 2.x application whose renderer is a React 18 + TypeScript single-page app built with Vite. app/src/App.tsx mounts the page-level router and provides the global layout, including the sidebar, top bar, and the theme provider that toggles between dark and light modes Source: app/src/App.tsx:1-80. The page tree registered at the top level resolves to five primary destinations: Dashboard, Search, Graph, Document Viewer, and Settings.

app/src/DashboardPage.tsx is the landing surface and aggregates corpus-level metrics: document count, embedding index size, recent ingestion events, and a small panel surfacing the state of any long-running background jobs Source: app/src/DashboardPage.tsx:1-60. app/src/SearchPage.tsx is the heaviest interactive surface and provides both a lexical and a hybrid query input. It submits user queries through Tauri's invoke channel to the Rust bridge, which proxies them into the Python backend Source: app/src/SearchPage.tsx:40-120.

app/src/GraphPage.tsx renders an entity-relationship visualisation powered by a force-directed layout. The graph data is fetched lazily and incrementally as the user pans and zooms, so the page can scale to thousands of nodes without blocking the renderer Source: app/src/GraphPage.tsx:20-95. app/src/DocumentViewer.tsx is responsible for rendering the original extracted content of a single document, including page-level metadata and inline thumbnails for image-embedded pages Source: app/src/DocumentViewer.tsx:1-50. app/src/SettingsPage.tsx exposes user-tunable parameters — index backend selection, BM25/HNSW weighting, embedding model identifier, and MCP server endpoint — all persisted through Tauri's secure store plugin Source: app/src/SettingsPage.tsx:30-140.

2. Tauri Rust Backend Bridge

app/src-tauri/src/main.rs is the entry point for the Rust process. It registers the application state, opens the embedded Python interpreter through PyO3, and forwards each #[tauri::command] invocation to the corresponding Python function. The commands are deliberately thin wrappers: they validate input, hand off to the Python core, and serialise the response back as JSON Source: app/src-tauri/src/main.rs:1-90.

app/src-tauri/tauri.conf.json declares the application identifier, the bundled binary target, the list of allowed APIs, and the CSP that locks the renderer down so it cannot load remote scripts Source: app/src-tauri/tauri.conf.json:1-120. The build configuration lives in app/package.json, which wires together the Vite dev server, the Tauri CLI, and the TypeScript compiler Source: app/package.json:1-60.

flowchart LR
    UI[React UI<br/>App.tsx] -->|invoke| Rust[Tauri Rust<br/>main.rs]
    Rust -->|PyO3| Core[Python Core]
    Core --> MCP[MCP Server<br/>server.py]
    MCP -->|stdio| Agent[External AI Agent]
    MCP -->|stdio| UI

3. The 30-Tool MCP Surface

mcp/server.py boots an MCP-compliant server over stdio. It registers 30 tools organised into six functional groups and performs capability negotiation with the connecting client at startup Source: mcp/server.py:1-80. The tool catalogue is exported from mcp/tools/__init__.py, which enumerates each tool with its name, JSON Schema, and a one-line description consumed by the MCP tools/list response Source: mcp/tools/__init__.py:1-60.

The six tool groups are:

• Corpus management — list_documents, get_document, delete_document, ingest_path, ingest_url.
• Search and retrieval — lexical_search, semantic_search, hybrid_search, bm25_hnsw_search.
• Index operations — reindex, compact_index, set_hnsw_params, get_index_stats.
• Graph traversal — get_entity, neighbours, shortest_path, subgraph.
• Summarisation and extraction — summarise, extract_entities, extract_relations, classify_document.
• System and observability — health, metrics, list_jobs, cancel_job, set_log_level, version.

Each tool accepts a JSON object validated against its declared schema and returns either a structured payload or a streaming response. Tools that may take longer than a few hundred milliseconds return a job handle so the agent can poll list_jobs and cancel_job rather than blocking the connection.

4. Operational Notes and Community Context

The v1.2.0 release introduced HNSW-backed approximate nearest-neighbour search, which is exposed to MCP clients through semantic_search, hybrid_search, and bm25_hnsw_search Source: release notes v1.2.0. The v2.0.0 release stabilised the 30-tool catalogue and made the schema version a first-class field returned by the version tool, so clients can detect breaking changes at handshake time Source: release notes v2.0.0.

Because the React UI and any MCP agent invoke the same Python functions, end-to-end behaviour is identical across both surfaces. Operators who prefer scripted control can disable the Tauri window and run the Python core with the MCP server alone, which is the recommended path for headless deployments and CI pipelines.

Related Pages

Related topics: Project Overview & Installation, Core Architecture: Retrieval, Fusion & Cognitive Layer, Tauri Desktop App & 30-Tool MCP Surface

Deployment, Security, Operations & Extensibility

DepthFusion exposes a layered operational surface that lets teams deploy the fused BM25/HNSW retrieval stack (HNSWStore, E-45) as a long-running service, secure every request with OIDC-backed identity, and extend the system through a generated CLI and CI matrix. This page documents the moving parts operators touch when shipping DepthFusion v2.0.0, as referenced in the v1.2.0 release notes which introduced the REST API service and hardening themes still present in the latest release.

Deployment Topology

DepthFusion ships as a Python package whose entry points are configured in pyproject.toml and exposed as a REST service plus a CLI. The deployment surface has three coordinated layers:

1. A REST API process backed by src/depthfusion/serving/rest_api.py, which exposes fused recall (BM25 + HNSW) over HTTP and is the primary runtime for production traffic.
2. A generated CLI in src/depthfusion/cli/generated_cli.py that drives the same recall and indexing code paths for operators and CI jobs.
3. A systemd unit defined in src/depthfusion/serving/systemd_unit.py, which supervises the REST process with restart, logging, and capability constraints appropriate for a server workload.

flowchart LR
    Client[Operator / CI] --> CLI[generated_cli.py]
    Client --> REST[rest_api.py :8080]
    REST --> IDX[HNSWStore<br/>hnswlib]
    REST --> BM25[BM25 fused recall]
    systemd[systemd_unit.py] -. supervises .-> REST
    OIDC[oidc_client.py] --> REST
    Registry[device_registry.py] --> REST

Operators typically install the package from pyproject.toml, enable the unit from systemd_unit.py, and front the REST listener with their reverse proxy of choice. The HNSWStore introduced for E-45 is index-resident in the REST process so the service can answer approximate nearest-neighbour queries without an external vector database. Source: src/depthfusion/retrieval/hnsw_store.py:1-120.

Security: Identity, Tokens, and Device Trust

Security is centralised in the identity package and follows a strict trust chain: every REST and CLI call resolves to a caller identity, which is then bound to a device record, a short-lived lease, and a validated token.

• src/depthfusion/identity/oidc_client.py performs the OAuth2/OIDC dance against the configured provider, exchanging client credentials for ID and access tokens and refreshing them on expiry. Source: src/depthfusion/identity/oidc_client.py:1-80.
• src/depthfusion/identity/token_validator.py verifies signature, issuer, audience, and expiry on each inbound request before the REST handler is invoked. Source: src/depthfusion/identity/token_validator.py:1-60.
• src/depthfusion/identity/device_registry.py keeps the authoritative list of enrolled device identifiers, and device_lease.py issues time-bound leases so a stolen token cannot be replayed indefinitely. Source: src/depthfusion/identity/device_lease.py:1-90.
• src/depthfusion/identity/device_keychain.py stores device-bound secrets (private keys, refresh tokens) in an OS-protected store, never on disk in plaintext.
• src/depthfusion/identity/service_account.py defines non-human identities used by the REST service and CI jobs, completing the ruflo-mod trust contract referenced in the v1.2.0 release.

This layered design means that adding a new caller class (for example, a new microservice) requires only a new service account entry and an enrolled device; no changes to the REST handlers are needed.

Operations: CI Matrix and Hardening

The operational story is anchored by the CI pipeline defined in .github/workflows/ci.yml. The v1.2.0 release notes call out "CI matrix hardening" and "security updates", which translate concretely into:

• A multi-OS, multi-Python matrix that exercises the REST entry point, the generated CLI, and the HNSWStore index round-trip on every pull request.
• Dependency-update automation that bumps hnswlib, OIDC client libraries, and HTTP server libraries so security advisories are addressed without manual triage.
• Static analysis and secret-scanning steps gated on the workflow file, so the same identity code (oidc_client.py, token_validator.py) is linted under the same rules as the recall stack.

For day-2 operations, the systemd unit from systemd_unit.py exposes Restart=, LimitNOFILE=, and journal logging, and the CLI in generated_cli.py exposes operational subcommands (index build, recall probe, lease rotation) so on-call engineers do not need raw HTTP to investigate issues.

Extensibility

DepthFusion is structured for plug-in recall and plug-in identity:

• New retriever backends (for example, alternative ANN indices) can be added alongside hnsw_store.py and wired into the fused recall path used by rest_api.py.
• New identity providers slot in behind oidc_client.py by conforming to its token interface, after which token_validator.py and device_registry.py need no changes.
• New CLI subcommands are emitted from the same internal command registry that produces generated_cli.py, so additions are picked up in both the operator-facing CLI and any REST endpoints generated from the same registry.

Together these surfaces make the v2.0.0 release operable as a hardened, identity-aware retrieval service that teams can extend without forking the core stack.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/gregdigittal/depthfusion

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/gregdigittal/depthfusion

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gregdigittal/depthfusion

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/gregdigittal/depthfusion

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/gregdigittal/depthfusion

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gregdigittal/depthfusion

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gregdigittal/depthfusion

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using depthfusion with real data or production workflows.

• v2.2.0 — Auth wizard & claims rectification - github / github_release
• v2.1.1 — ChatGPT Desktop MCP Integration + Security Hardening - github / github_release
• DepthFusion v2.0.0 - github / github_release
• v1.2.0 — CI matrix hardening, security updates, REST API service - github / github_release
• Configuration risk requires verification - GitHub / issue