Related Pages

Related topics: Architecture, Watcher Pipeline, and Bitemporal Graph, MCP Tools, Hierarchical Clusters, and AI Integration, Context Cost Telemetry, Deployment, Frontend, and Operations

Introduction and Installation

memex is an open-source memory engine and MCP (Model Context Protocol) server that gives LLM agents a durable, queryable long-term memory layer. The project ships both a Python package and an NPM-distributed MCP server (current latest release: v0.5.1), so it can be embedded either as a library inside an agent runtime or launched as a standalone MCP process that any MCP-compatible client (Claude Desktop, Cursor, etc.) can attach to. Source: npm/package.json:1-30

The v0.4.0 release introduced Context Cost Telemetry and Confidence-Weighted Write Discipline, signaling that the project treats memory writes as a budgeted, scored operation rather than a free-form append. Source: README.md:1-80 This makes installation correctness especially important, because bad configuration will distort the telemetry the tool is designed to surface.

What memex Does

At a high level, memex sits between an LLM agent's context window and a persistent store. It exposes a small set of memory primitives (write, read, search, prune, summarize) over MCP, so the agent can offload facts, preferences, and working notes without re-prompting them on every turn. Source: server.json:1-40

The Python distribution in pyproject.toml provides the same primitives as an importable library for embedding directly into custom agent loops. Source: pyproject.toml:1-60

System Requirements

Because the project has both a Python and a Node distribution, requirements are split:

Source: pyproject.toml:1-60, Source: npm/package.json:1-30

Installing the Python Package

The Python side is a standard PEP 517 / PEP 621 package declared in pyproject.toml. From a fresh clone:

1. Clone the repository.
2. From the project root, install in editable mode so local edits are picked up:

pip install -e .

This installs the memex library plus its declared runtime dependencies. Source: pyproject.toml:1-60

If you only want to consume the library without cloning, the same project metadata should make it publishable to a standard index, in which case pip install memex is sufficient.

Installing the NPM MCP Server

The MCP server lives under npm/ and is the primary integration path for desktop clients. Installation from source:

cd npm
npm install
npm link          # makes the `memex-mcp` binary available globally

The bin field in npm/package.json registers memex-mcp (implemented by npm/bin/memex-mcp.js) as the executable entry point. Source: npm/package.json:1-30, Source: npm/bin/memex-mcp.js:1-60

For end users who don't want to clone, the package can also be installed directly from the registry once published, and memex-mcp added to the MCP client's server.json configuration. The repository ships a reference server.json at the project root describing the server's launch command, arguments, and metadata fields accepted by MCP host applications. Source: server.json:1-40

Wiring memex into an MCP Client

After installation, point your MCP host (e.g. Claude Desktop, Continue, Cursor) at the memex-mcp binary. The typical flow is:

1. Locate the host's server.json (or equivalent MCP config).
2. Add an entry that references either the globally linked memex-mcp binary or an absolute path to npm/bin/memex-mcp.js. Source: server.json:1-40
3. Restart the host so it spawns the MCP process and discovers the exposed memory tools.

The npm/README.md is the canonical, client-facing install guide and should be consulted for host-specific snippets (Claude Desktop path locations, npx fallbacks, etc.). Source: npm/README.md:1-80

Quick Sanity Check

Once connected, a minimal smoke test is to ask the agent to:

1. Write a small memory item ("remember that the user prefers dark mode").
2. Read it back in a fresh session.

If the value persists, the storage backend is wired correctly and the write pipeline — including the confidence weighting introduced in v0.4.0 — is functioning. Source: npm/README.md:1-80

Choosing the Right Install Path

• Building a Python agent? → install pip install -e . and import memex directly.
• Driving an existing MCP-compatible client? → install the NPM package and register memex-mcp via server.json.
• Packaging or forking the server? → edit npm/bin/memex-mcp.js and bump version in npm/package.json.

In all three cases, keep an eye on the release notes between minor versions: v0.4.0 changed how writes are scored, which can subtly alter behavior for existing memories after upgrade. Source: README.md:1-80

Related Pages

Related topics: Introduction and Installation, MCP Tools, Hierarchical Clusters, and AI Integration, Context Cost Telemetry, Deployment, Frontend, and Operations

Architecture, Watcher Pipeline, and Bitemporal Graph

1. High-Level Architecture

memex is organized as a layered system in which a long-running watcher daemon observes a local workspace and the surrounding Git repository, normalizes raw signals into typed events, and writes them into a bitemporal graph that downstream query and reasoning components can consume.

The watcher subsystem is split into small, single-purpose modules so that each ingestion source (filesystem, Git commits, Git hooks) can be tested in isolation while sharing a common event vocabulary.

The release notes for v0.4.0 emphasize *Context Cost Telemetry* and *Confidence-Weighted Write Discipline*, both of which are enforced at the boundary between the watcher pipeline and the bitemporal graph so that noisy or low-confidence events do not pollute the historical record.

Source: memex/watcher/daemon.py:1-40

2. Watcher Pipeline

The pipeline is a unidirectional data flow: raw signal → source adapter → typed event → router → graph writer.

2.1 Filesystem Observer

fs_observer.py wraps a watchdog-style observer and converts native filesystem callbacks into FSEvent instances. It debounces rapid edits, normalizes paths to repository-relative form, and emits one event per logical change rather than per syscall.

Source: memex/watcher/fs_observer.py:1-60

2.2 Git Commit Poller

commit_poller.py periodically queries git log (or an equivalent libgit2 binding) for new commits and produces CommitEvent records. Because polling is pull-based, it acts as the safety net for events missed by the hook path.

Source: memex/watcher/commit_poller.py:1-80

2.3 Git Hook Adapter

git_hook.py exposes an entry point invoked from Git's post-commit and post-checkout hooks. Hook events arrive with the highest fidelity and lowest latency, so the router treats them as the preferred source when present, falling back to the poller for missed writes.

Source: memex/watcher/git_hook.py:1-50

2.4 Event Router

event_router.py is the single fan-in point. It deduplicates events by (path, hash, ts), attaches a confidence score (the basis of the v0.4.0 *Confidence-Weighted Write Discipline*), and forwards accepted events to the bitemporal writer while dropping or down-weighting the rest. Cost telemetry counters are incremented here so that the daemon can report context-window pressure to operators.

Source: memex/watcher/event_router.py:1-120

3. Event Schema

events.py defines the canonical dataclasses — FSEvent, CommitEvent, HookEvent — each carrying a stable identifier, a source kind, a timestamp, and an optional payload. Using a single schema lets the router apply uniform rules regardless of which adapter produced the event.

Source: memex/watcher/events.py:1-90

4. Bitemporal Graph

The graph store is bitemporal: every fact is stored along two independent time axes.

• Valid time (vt_from / vt_to) — when the change is *true in the world* (e.g., when the file was actually edited).
• Transaction time (tt_from / tt_to) — when memex *learned about* the change.

This dual-axis model is what lets the system answer questions like *“What did the codebase look like on Tuesday, and what did memex know about it on Wednesday?”* — a property that single-stamp stores cannot provide.

Writes are append-only. Corrections produce a new edge that supersedes the prior one rather than mutating history, which keeps the graph auditable. The router’s confidence weights feed a derived edge attribute that query layers can use to rank or filter results without rewriting the underlying record.

flowchart LR
    FS[fs_observer.py] -->|FSEvent| R[event_router.py]
    CP[commit_poller.py] -->|CommitEvent| R
    GH[git_hook.py] -->|HookEvent| R
    R -->|typed event| G[(Bitemporal Graph)]
    R -.->|telemetry| D[daemon.py]
    D -->|metrics| Op[Operator]

5. Daemon Coordination

daemon.py owns process lifecycle: it starts each source adapter on a controlled schedule, exposes a health endpoint for the cost-telemetry counters introduced in v0.4.0, and applies back-pressure by pausing adapters when the router’s pending queue exceeds a threshold. On shutdown it flushes in-flight events so that no transaction-time boundary is lost.

Source: memex/watcher/daemon.py:40-120

6. Operational Notes

• Deduplication is keyed on (source, path, content_hash) in event_router.py to prevent the poller and the hook from double-writing the same commit.
• Confidence-Weighted Write Discipline (v0.4.0) means low-confidence events are stored with reduced priority rather than being silently dropped, preserving the audit trail while biasing downstream retrieval toward higher-signal edges.
• Context Cost Telemetry (v0.4.0) is exported by the daemon so that operators can correlate graph growth with retrieval-budget consumption in the latest v0.5.1 release.

Source: memex/watcher/event_router.py:60-140, memex/watcher/daemon.py:80-160

Related Pages

Related topics: Architecture, Watcher Pipeline, and Bitemporal Graph, Context Cost Telemetry, Deployment, Frontend, and Operations

MCP Tools, Hierarchical Clusters, and AI Integration

Overview

The memex/mcp_server package exposes the memex memory system to AI clients over the Model Context Protocol (MCP). It groups its capabilities into four tool modules — read, write, impact, and explain — that share a single query layer and a hierarchical cluster schema stored in the underlying vector/graph index. The server entry point registers these tools and forwards calls to the appropriate handler. Source: memex/mcp_server/server.py:1-80

The MCP layer is intentionally thin: each tool validates input, calls into queries.py, formats the response, and returns JSON that an AI agent can consume directly. Source: memex/mcp_server/server.py:40-120

Hierarchical Cluster Model

Memex organizes memories into a multi-level hierarchy of clusters rather than a flat tag system. Each cluster has a centroid embedding, a representative label, a list of member node IDs, and optional parent/child pointers that form a tree.

• Cluster creation is driven by embedding similarity during upsert_node; new nodes are either assigned to an existing cluster above a similarity threshold or seed a new one. Source: memex/mcp_server/tools_write.py:60-140
• Cluster promotion/demotion happens in recluster, which merges under-populated clusters and splits over-saturated ones using the same similarity threshold. Source: memex/mcp_server/tools_write.py:160-210
• Cluster reads return members, centroid, and a breadcrumb path to the root, which is what expand uses to walk up and down the tree. Source: memex/mcp_server/tools_read.py:90-160

flowchart TD
    A[Root Cluster] --> B[Topic: Architecture]
    A --> C[Topic: Decisions]
    B --> B1[Sub: API Design]
    B --> B2[Sub: Data Model]
    B1 --> N1[Node: endpoint spec]
    B1 --> N2[Node: auth flow]
    B2 --> N3[Node: schema v2]
    C --> N4[Node: ADR-007]

This hierarchy is the substrate the AI tools reason over: explain traverses upward to gather context, while impact walks downward to find affected nodes. Source: memex/mcp_server/tools_explain.py:30-95

Read and Write Discipline

Read and write are deliberately separated so the AI agent can be given read-only access in some deployments.

Read path (tools_read.py) exposes:

• get_node(id) — full record with edges and cluster membership. Source: memex/mcp_server/tools_read.py:1-60
• get_cluster(id, depth=2) — subtree within a configurable depth. Source: memex/mcp_server/tools_read.py:90-160
• search(query, k=10) — vector search scoped to a cluster subtree when provided. Source: memex/mcp_server/tools_read.py:180-230

Write path (tools_write.py) enforces confidence-weighted write discipline, introduced in v0.4.0 alongside context-cost telemetry:

• Each write returns a confidence score and a context_cost estimate; callers can opt to suppress low-confidence writes. Source: memex/mcp_server/tools_write.py:1-60
• upsert_node requires at least one supporting edge or a cluster assignment; orphan nodes are rejected. Source: memex/mcp_server/tools_write.py:80-130
• link validates that both endpoints exist and that the proposed edge type is in the allowed set. Source: memex/mcp_server/tools_write.py:140-180

Community feedback around v0.4.0 emphasized that confidence gating prevents AI agents from polluting the memory graph with speculative facts; this is reflected in the strict validation in the write tools. Source: memex/mcp_server/tools_write.py:200-260

Impact and Explain (AI-Facing Tools)

These two modules are the primary surface AI agents use to *reason* about the memory graph.

Impact tools (tools_impact.py) answer "what breaks if X changes?":

• impact_radius(node_id, max_hops=3) — returns the set of nodes reachable within N hops, weighted by edge type. Source: memex/mcp_server/tools_impact.py:30-100
• dependents(node_id) — fan-in only, useful for "who relies on this" queries. Source: memex/mcp_server/tools_impact.py:120-170
• critical_path(node_a, node_b) — shortest dependency chain between two nodes. Source: memex/mcp_server/tools_impact.py:190-240

Explain tools (tools_explain.py) translate graph data into prose:

• explain_node(id) — combines node content, incoming/outgoing edges, and cluster summary into a single narrative. Source: memex/mcp_server/tools_explain.py:1-60
• summarize_cluster(id) — produces a roll-up description by sampling member nodes and the centroid label. Source: memex/mcp_server/tools_explain.py:80-130
• narrate_diff(before_id, after_id) — compares two versions of a node and outputs a human-readable diff. Source: memex/mcp_server/tools_explain.py:150-200

Both modules share queries.py, which centralizes the SQL/vector queries so that caching and pagination are consistent across tools. Source: memex/mcp_server/queries.py:1-90

How They Fit Together

A typical AI-agent session flows: search to find candidate nodes → expand to pull a cluster subtree → explain_node for the most relevant hits → impact_radius before proposing any change → upsert_node / link with a confidence score, gated by the v0.4.0 write discipline. The hierarchical cluster schema is what makes expand cheap and what gives summarize_cluster a coherent scope. The MCP server in server.py is the single registration point that exposes all of the above to MCP-compatible clients such as Claude, IDE agents, or custom orchestrators. Source: memex/mcp_server/server.py:80-200

This separation — thin transport, shared query layer, four tool categories, and a hierarchical cluster model — is the core of memex's AI integration story and the reason it can be wired into an agent loop with minimal glue code.

Related Pages

Related topics: Architecture, Watcher Pipeline, and Bitemporal Graph, MCP Tools, Hierarchical Clusters, and AI Integration

Context Cost Telemetry, Deployment, Frontend, and Operations

The Memex project bundles four cross-cutting concerns into its v0.4.x line: Context Cost Telemetry (how much LLM budget a memory write or read actually consumes), Deployment (the CLI entry points that ship the project), Frontend (the graph-rendering and templating layer presented to operators), and Operations (the review/discipline workflows that gate low-confidence writes). This page walks through each concern, ties them to specific source files, and surfaces the design intent behind the v0.4.0 "Confidence-Weighted Write Discipline" release.

1. Context Cost Telemetry

The telemetry subsystem is split into two cooperating modules. memex/graph/telemetry.py instruments graph operations so every read, write, or recall emits a structured event containing token counts, latency, and the confidence score returned by the underlying model. memex/graph/stats.py aggregates those events into rolling statistics — mean cost per write, p95 recall cost, and total tokens burned per project — that are surfaced both to the CLI and to any external dashboard sink configured at deployment time.

Telemetry is designed to be pull-friendly: callers can query the latest snapshot without waiting for a flush, which keeps the CLI responsive during interactive review sessions. The aggregator keys events by operation type (write, recall, summarize) and by graph namespace, so operators can spot a single noisy subgraph that dominates the budget.

Source: memex/graph/telemetry.py:1-120 Source: memex/graph/stats.py:1-140

2. Deployment via the CLI Surface

memex/cli.py is the deployment anchor: it parses arguments, configures the telemetry backend, wires the graph store, and dispatches to the sub-command modules. Sub-commands are deliberately thin so that cli.py remains the single source of truth for global flags such as --telemetry-sink, --cost-budget, and --confidence-floor. These flags map directly onto the telemetry and stats modules — for example, --cost-budget is enforced by reading the rolling aggregate produced by stats.py before each write.

The CLI doubles as the operational contract. Operators invoke memex ... on a host or container and pass through environment variables for the graph backend, the LLM provider, and the telemetry export target. Because all sub-commands route through cli.py, swapping the deployment target (local workstation, CI runner, or container) does not require re-tooling the frontend or the review workflow.

Source: memex/cli.py:1-180

3. Frontend: Graph Rendering and Templates

The frontend layer is intentionally text-first. memex/cli_graph.py renders the memory graph to the terminal — typically a compact adjacency summary plus the most recent write events — so operators can verify state without launching a separate UI. Rendering is driven by a small set of formatters that respect terminal width and respect the --quiet flag for CI usage.

memex/cli_graph_template.py provides the template scaffolding that cli_graph.py fills in. Templates separate *layout* (header, node blocks, edge table, cost footer) from *content* (the actual nodes and the telemetry footer). This separation is what makes the cost telemetry visible by default: the footer template always queries stats.py and embeds the current session's token usage, so a memex graph show command is also an implicit telemetry probe.

Source: memex/cli_graph.py:1-160 Source: memex/cli_graph_template.py:1-120

4. Operations: Confidence-Weighted Write Discipline

memex/cli_review.py operationalizes the v0.4.0 release's headline feature. Each candidate write produced by the graph layer carries a confidence score from the upstream model plus the cost estimate from telemetry.py. The review command applies the configured confidence-floor and either promotes the write, queues it for human review, or discards it. This is the "Confidence-Weighted Write Discipline" referenced in the v0.4.0 release notes.

In practice, the operations loop looks like:

1. cli.py receives a write intent.
2. telemetry.py estimates the token cost.
3. stats.py checks remaining budget.
4. cli_review.py enforces the confidence floor and routes the write.
5. cli_graph.py re-renders the affected subgraph with the new cost footer.

Because every step emits a telemetry event, an operator can later replay a session and answer two questions that are otherwise hard: "how much did this write cost?" and "did the confidence justify the cost?" Those answers flow back into the next round of discipline tuning.

Source: memex/cli_review.py:1-200 Source: memex/cli.py:60-140 Source: memex/graph/telemetry.py:80-160

Data Flow at a Glance

The table below summarizes how each file participates in the loop and which artifact it consumes or produces.

Together these six files implement a closed loop: every write is measured, every measurement feeds a budget, and every budget decision is reviewable through the same CLI surface that ships the project. That loop is the core of the v0.4.x "Context Cost Telemetry & Confidence-Weighted Write Discipline" story and is the operational contract the latest release (v0.5.1) continues to evolve.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/STiFLeR7/memex

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/STiFLeR7/memex

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/STiFLeR7/memex

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/STiFLeR7/memex

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/STiFLeR7/memex

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/STiFLeR7/memex

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/STiFLeR7/memex

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using memex with real data or production workflows.

• v0.4.0 - github / github_release
• Configuration risk requires verification - GitHub / issue