Related Pages

Related topics: MCP Tools, Setup Wizard & AI Client Configuration, Knowledge Management, Data Model & Version Chains, Operations, Deployment, Governance & Cross-Tool Continuity

Overview & System Architecture

Purpose and Scope

Piia Engram is a local-first memory and continuity layer designed for AI coding agents (notably Claude Code and compatible MCP clients). It provides durable, sensitivity-screened knowledge storage — composed of *lessons* (learned practices) and *decisions* (architectural choices) — that persists across sessions and across different tools, so an agent can resume work without losing accumulated context.

The project is organized around four architectural concerns:

1. Knowledge lifecycle — staging → verified promotion, sensitivity classification, and review.
2. Client integration — Claude Code hooks for automatic absorption, resume, and save.
3. Governance and observability — audit logs, beta event tracking, and optional telemetry.
4. Export and portability — AGENTS.md / CLAUDE.md blocks, backup/restore, and cross-tool reconciliation.

The latest release line (v3.49.0) emphasizes cross-tool continuity, introducing opt-in import version-chain materialization for engram import --apply --yes --materialize-version-chain and a new MCIC v1 contract, as described in the community release notes.

High-Level Architecture

The following diagram summarizes the major subsystems and how they relate during a typical session.

flowchart LR
  Client[AI Client<br/>Claude Code / MCP] -->|hooks| Hooks
  Hooks[Hook Layer<br/>SessionStart / Stop / PostCompact] --> Store
  Store[(Local Store<br/>~/.engram/)] --> Core[Core Engine<br/>lessons + decisions]
  Core --> Sensitivity[Sensitivity Classifier]
  Core --> Audit[Audit Logger]
  Core --> Beta[Beta Event Tracker]
  Core --> Export[AGENTS.md Export]
  Core -.telemetry.-> Worker[Cloudflare Worker + D1]
  Worker --> Dashboard[Public Dashboard]
  Export --> RepoFiles[AGENTS.md / CLAUDE.md]
  Store -.backup.-> Import[Import / Restore]
  Import --> Core

Source: src/piia_engram/hooks/auto_inject_resume_brief.py, src/piia_engram/hooks/auto_save_on_stop.py, src/piia_engram/audit.py, src/piia_engram/beta_tracker.py, src/piia_engram/agents_md_export.py, worker/src/index.js

Key Subsystems

Hook Layer (Client Integration)

Three Claude Code hook entry points drive the lifecycle without requiring manual intervention:

• SessionStart — calls engram.get_resume_brief(...) with a token_budget and emits a hookSpecificOutput.additionalContext Markdown brief. The hook exits silently (prints {"continue": true}) if no brief is available or on any exception, so it never blocks the host. Source: src/piia_engram/hooks/auto_inject_resume_brief.py
• Stop — parses the session transcript, collects invoked tool names, and persists the session if a minimum message threshold is met. The threshold is env-configurable so that PreCompact can lower it. Source: src/piia_engram/hooks/auto_save_on_stop.py
• PostCompact — absorbs the compact summary into the store with event_type="compact". As of v3.31, semantic extraction (lessons/decisions) is delegated to a sibling agent hook to prevent double-writes into the staging tier. Source: src/piia_engram/hooks/auto_absorb_compact.py

Knowledge Store and Sensitivity Gating

Knowledge entries carry an explicit tier field. The export subsystem is *fail-closed*: an entry must be explicitly tier == "verified" and status == "active" to be eligible for AGENTS.md emission. A missing tier is treated as not-exportable, so a future writer that forgets to stamp the tier cannot leak into the export. Sensitivity classification is delegated to sensitivity.classify_item, and entries above the caller's max_sensitivity ceiling (default "work") are excluded. Source: src/piia_engram/agents_md_export.py

Governance and Audit

Every identity and knowledge read/write is recorded to ~/.engram/audit.log in JSON-lines format. Read operations are additionally gated by governance_runtime.governance_enabled() and caller_is_owner(root) — non-owner reads are silently skipped, preventing audit pollution from other users on a shared host. Source: src/piia_engram/audit.py

A separate beta_tracker.py records governance lifecycle *metadata only* (no knowledge content) to ~/.engram/beta_events.jsonl. Tracked events include knowledge_created, knowledge_promoted, knowledge_reviewed, knowledge_rejected, cold_start, session_end, and reconcile. Tracking is on by default and disabled by setting ENGRAM_BETA_TRACKING=0. Source: src/piia_engram/beta_tracker.py

Telemetry and Feedback (Cloudflare Worker)

An opt-in telemetry path POSTs anonymized daily rollups to a Cloudflare Worker backed by D1. The worker performs a *tiered INSERT* — attempting v1.1 (P0 + P1) columns first, falling back to v1 (P0) if the schema is not yet migrated, and finally to a legacy insert — so that no event is ever dropped during schema rollouts. It also exposes a feedback endpoint with payload size limits and serves a public dashboard rendered from aggregated stats. Source: worker/src/index.js

Client Validation Harness

A separate client_validation.py module defines the evidence contract for live AI client runs (e.g., Hermes CLI, OpenClaw file bridge). It lists required artifacts (run_meta.json, tool_locations.json, timings.json, zero_pollution.txt, REPORT.md, etc.) and the keys that run_meta.json must contain (client_id, client_version, surface, model, engram_mode, environment_arm, workspace_isolated, home_isolated, write_tools_allowed, known_limitations). This keeps simulated memory cycle tests distinct from real-client validation. Source: src/piia_engram/client_validation.py

Community Context

Issue #8 surfaced a recurring theme in the project's ecosystem discussion: third-party API gateway providers (e.g., ddshub.cc) frequently approach MCP authors seeking cross-promotion. This is informational rather than architectural — Engram itself does not depend on or endorse any specific API gateway. The architecture remains local-first, with telemetry strictly opt-in and schema-evolution-tolerant.

See Also

• Knowledge Tiers and Promotion Lifecycle
• AGENTS.md / CLAUDE.md Export Rules
• Claude Code Hook Reference
• Telemetry and Privacy Model
• Backup, Import, and Version Chains (v3.49.0)

Related Pages

Related topics: Overview & System Architecture, Knowledge Management, Data Model & Version Chains, Operations, Deployment, Governance & Cross-Tool Continuity

MCP Tools, Setup Wizard & AI Client Configuration

Overview

Engram exposes its memory layer to AI clients through three cooperating layers:

1. MCP Server — registers the agent-facing tool surface (add_lesson, add_decision, get_user_context, wrap_up_session, get_resume_brief, etc.) over the Model Context Protocol.
2. Setup Wizard — a guided first-run flow that initializes ~/.engram/, registers the server with host tools, and configures per-client hooks.
3. AI Client Configuration — Claude Code hooks, AGENTS.md/CLAUDE.md export, and client validation evidence that wire Engram into real editors and harnesses.

The MCP surface is the contract; the setup wizard is the on-ramp; the client configuration is the integration glue that makes the surface useful in a real session.

flowchart LR
    Client[AI Client<br/>Claude Code / Cursor / Hermes]
    Hook[Client Hooks<br/>auto_inject_resume_brief<br/>auto_absorb_compact<br/>auto_save_on_stop]
    MCP[MCP Server<br/>mcp_server.py]
    Wizard[Setup Wizard<br/>setup_wizard.py]
    Store[(~/.engram/<br/>knowledge, identity, audit)]
    Client -->|stdio| MCP
    Client -->|JSON hook| Hook
    Wizard -.->|configures| MCP
    Wizard -.->|configures| Hook
    MCP --> Store
    Hook --> Store
    Hook -->|additionalContext| Client

MCP Tool Surface

The MCP server is the only agent-facing API. It is registered separately from export and validation logic so no new tool ships without an explicit wiring step.

Tool categories registered by mcp_server.py (sourced from the module's registration and the public docs referenced in the codebase):

Every successful add_lesson / add_decision is instrumented by beta_tracker.track_event under the knowledge_created event; promotions emit knowledge_promoted; reviews emit knowledge_reviewed; deletes emit knowledge_rejected. Source: src/piia_engram/beta_tracker.py.

Read operations are gated by the audit log: when governance is enabled, AuditLogger.log returns early if the caller is not the owner of the store root. Source: src/piia_engram/audit.py:21-44.

Context and Recall Layer

Two complementary modules back the read tools:

• context.py / contexts.py — assemble a "user context" payload (identity + verified lessons + verified decisions) under sensitivity ceilings, project scoping, and token budgets.
• recall.py / recall_service.py — perform the actual retrieval (key match, semantic match, recency boost) and return ranked candidates that the context layer then formats.

get_resume_brief is the lighter cousin of get_user_context: it returns a Markdown block sized for a system-prompt splice (~1,500 tokens by default in the hook path). Source: src/piia_engram/hooks/auto_inject_resume_brief.py:1-26.

Setup Wizard

The setup wizard runs on first invocation (or when ~/.engram/ is missing) and is the recommended path for new users. It is responsible for:

• Creating the on-disk store root (~/.engram/ by default, overridable via ENGRAM_DIR). Source: src/piia_engram/beta_tracker.py:36-42.
• Selecting the target host (Claude Code, Cursor, Codex CLI, generic MCP client).
• Writing the host-specific MCP config snippet (e.g. .mcp.json for Claude Code).
• Registering the three Claude Code hooks (SessionStart, PreCompact/PostCompact chain, Stop).
• Seeding the identity profile and sensitivity ceiling.
• Emitting an AGENTS.md / CLAUDE.md starter block via the verified-only export.

The export helper that the wizard can pre-fill is strictly metadata-only: it filters to tier == "verified", status == "active", drops items above the caller's max_sensitivity ceiling (default work), and renders only the human-facing summary/choice — never ids, access counts, risk flags, or provenance internals. Source: src/piia_engram/agents_md_export.py:1-19.

sequenceDiagram
    participant U as User
    participant W as Setup Wizard
    participant FS as ~/.engram/
    participant C as AI Client
    U->>W: engram init
    W->>FS: create root, identity, tiers
    W->>U: pick host(s)
    W->>C: write .mcp.json + hooks
    W->>U: emit AGENTS.md export
    U-->>C: restart / reload

AI Client Configuration

Claude Code Hooks

Three hooks implement the "last mile" cross-session continuity. They are invoked by Claude Code, not by the MCP server.

A re-entry guard prevents recursive agent invocations: when CLAUDE_INVOKED_BY=engram_recursive is set, the SessionStart hook exits silently. Source: src/piia_engram/hooks/auto_inject_resume_brief.py:7-11.

The Stop hook respects an env-tunable flush threshold and a derived min_save_threshold = max(4, _flush_threshold() // 2) so PreCompact can lower the bar for the next attempt. Source: src/piia_engram/hooks/auto_save_on_stop.py:18-25.

AGENTS.md / CLAUDE.md Export

build_agents_md_export is a pure function over already-loaded entries — it does not read or write the store, and it does not register an MCP tool. Wiring it into an agent-facing surface is a separate, reviewed step. This is by design: no new agent-facing capability ships without an explicit wiring commit. Source: src/piia_engram/agents_md_export.py:5-19.

The export supports two scopes: scope="global" emits only non-project (generalizable) knowledge; scope="project" emits knowledge for a named project. Either way, sensitivity-screened items are dropped via sensitivity.classify_item and the SENSITIVITY_ORDER ranking. Source: src/piia_engram/agents_md_export.py:21-31.

Client Validation Evidence

client_validation.py is the evidence scaffolding for live client runs (e.g. Hermes CLI, OpenClaw file bridge) — intentionally separate from the simulated continuity_harness. It defines:

• A fixed EVIDENCE_FILES list a run must produce (run_meta.json, tool_locations.json, prompts/, raw/, parsed/, timings.json, zero_pollution.txt, REPORT.md, OPTIMIZATION_NOTES.md).
• A REQUIRED_RUN_META_KEYS whitelist (client_id, client_version, surface, model, engram_mode, environment_arm, workspace_isolated, home_isolated, write_tools_allowed, known_limitations).
• A FileSnapshot dataclass for zero-pollution comparisons (path, exists, sha256, size, mtime).
• A _LEVEL_ORDER ladder (L0–L5) for capability levels.

This is what allows a run to be reproduced, diffed, and graded without trusting the client. Source: src/piia_engram/client_validation.py:1-49.

Telemetry and Beta Tracking

By default, beta tracking writes one JSON line per governance event to ~/.engram/beta_events.jsonl. Set ENGRAM_BETA_TRACKING=0 to disable. The Cloudflare Worker (worker/src/index.js) ingests these events into a D1 database and serves a public dashboard plus a /v1/stats JSON API. The worker uses a tiered INSERT with graceful column fallback so schema migrations never drop events. Source: src/piia_engram/beta_tracker.py:44-55 and worker/src/index.js:1-60.

Common Failure Modes

See Also

• Knowledge Tiers & Governance Lifecycle
• Sensitivity Model & Re-exports
• Cross-Tool Continuity (Hermes / OpenClaw)
• Telemetry & Beta Event Schema
• MCIC v1 (released in v3.49.0)

Related Pages

Related topics: Overview & System Architecture, MCP Tools, Setup Wizard & AI Client Configuration, Operations, Deployment, Governance & Cross-Tool Continuity

Knowledge Management, Data Model & Version Chains

Overview

Engram's knowledge subsystem is the durable memory layer that survives across coding sessions and tool boundaries. It models two primary artifact types — lessons (what the user has learned) and decisions (what the user has chosen) — and governs their flow from initial capture through promotion and selective export. The data model is deliberately conservative: staging content never escapes by default, and only verified, non-sensitive knowledge is eligible for cross-tool projection such as AGENTS.md or CLAUDE.md. Source: src/piia_engram/agents_md_export.py:11-23

The lifecycle is instrumented end-to-end by an opt-out beta tracker that emits discrete events (knowledge_created, knowledge_promoted, knowledge_reviewed, knowledge_rejected, cold_start, session_end, reconcile) to ~/.engram/beta_events.jsonl, giving operators a queryable view of governance state without ever writing knowledge content itself. Source: src/piia_engram/beta_tracker.py:1-25

Data Model

Knowledge items are normalized into a small set of fields that drive every downstream filter:

flowchart LR
  A[Capture<br/>add_lesson / add_decision] -->|tier=staging| B[(Staging Store)]
  B -->|review| C{tier=verified}
  C -->|status=active<br/>sens ≤ work| D[Export Pool]
  D -->|scope=global| E[AGENTS.md / CLAUDE.md]
  D -->|scope=project| F[Project-scoped Export]
  C -->|rejected| G[(Archived)]
  B -->|reconcile| H[Version Chain]
  H --> D

Every read or write of identity and knowledge resources is mirrored to ~/.engram/audit.log in JSON-lines format, with read actions additionally gated by governance ownership checks before the entry is persisted. Source: src/piia_engram/audit.py:1-15, 39-48

Knowledge Lifecycle

The lifecycle is a pipeline of explicit, observable transitions rather than a free-form document store. Knowledge is created into staging via add_lesson or add_decision, with the originating source_tool and domain recorded at write time. Source: src/piia_engram/beta_tracker.py:15-17, 50-62

Promotion from staging to verified can occur through manual review or auto-promotion, both of which emit a knowledge_promoted event carrying a count and a method discriminator used for promotion-method aggregation in the telemetry worker. Source: src/piia_engram/beta_tracker.py:64-74

Hooks bridge session boundaries:

• auto_inject_resume_brief.py reads the verified store on SessionStart and emits a token-budgeted Markdown brief (default 1500 tokens) for cold-start context. Source: src/piia_engram/hooks/auto_inject_resume_brief.py:1-30
• auto_save_on_stop.py flushes session content on Stop, gated by a configurable message threshold and timestamp parsing. Source: src/piia_engram/hooks/auto_save_on_stop.py:18-67
• auto_absorb_compact.py ingests compaction events; semantic extraction into lessons/decisions is delegated to a sibling PostCompact agent hook to avoid double-writing the staging tier. Source: src/piia_engram/hooks/auto_absorb_compact.py:1-25

Cold-start invocation and session wrap-up each emit their own cold_start and session_end events, and the reconcile event captures cross-tool memory synchronization. Source: src/piia_engram/beta_tracker.py:11-15

Version Chains

Version chains are Engram's mechanism for preserving reviewed conflicts between same-key items during import, rather than silently overwriting them. v3.49.0 introduces an opt-in flag — engram import <backup.json> --apply --yes --materialize-version-chain — that materializes these conflicts as explicit version-chain entries; the default import path remains conservative and does not enable this behavior. Source: community release notes (piia-engram v3.49.0).

A version-chain entry is therefore the third state an item can take after staging and verified: it represents a deliberate, human-reviewed divergence point, typically surfaced when reconciling two stores (e.g., a local checkpoint and a remote backup) that disagree on the same key. Because materialization is opt-in, the safety properties of the default import — verified-only projection, sensitivity screening via classify_item, and metadata-only export — are preserved when the flag is omitted. Source: src/piia_engram/agents_md_export.py:15-23, 120-145

Operators can confirm the chain is intact by inspecting beta_events.jsonl for reconcile events carrying the relevant daily_id and version, and by reviewing audit log entries tagged with action import and the affected resource path. Source: src/piia_engram/audit.py:18-30

See Also

• AGENTS.md Export & Cross-Tool Continuity
• Hooks: SessionStart / Stop / Compact
• Beta Telemetry & Audit Logging
• Import / Export & Backup Workflows

Related Pages

Related topics: Overview & System Architecture, MCP Tools, Setup Wizard & AI Client Configuration, Knowledge Management, Data Model & Version Chains

Doramagic Pitfall Log

Found 20 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Security or permission risk - Security or permission risk requires verification.

1. Security or permission risk: Security or permission risk requires verification

• Severity: high
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/Patdolitse/piia-engram/issues/23

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: [Bug] 'engram setup' deletes users existing configs without backups.
• User impact: Developers may fail before the first successful local run: [Bug] 'engram setup' deletes users existing configs without backups.
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: [Bug] 'engram setup' deletes users existing configs without backups.. Context: Observed when using python, macos
• Evidence: failure_mode_cluster:github_issue | https://github.com/Patdolitse/piia-engram/issues/23

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

4. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: v3.45.2 - CI entry-point patch
• User impact: Upgrade or migration may change expected behavior: v3.45.2 - CI entry-point patch
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.45.2 - CI entry-point patch. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.45.2

5. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: Engram v3.47.0
• User impact: Upgrade or migration may change expected behavior: Engram v3.47.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: Engram v3.47.0. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.47.0

7. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.45.1 - CI packaging patch
• User impact: Upgrade or migration may change expected behavior: v3.45.1 - CI packaging patch
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.45.1 - CI packaging patch. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.45.1

8. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.46.0
• User impact: Upgrade or migration may change expected behavior: v3.46.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.46.0. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.46.0

9. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.47.1 - Public Truth Sync
• User impact: Upgrade or migration may change expected behavior: v3.47.1 - Public Truth Sync
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.47.1 - Public Truth Sync. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.47.1

10. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.48.0
• User impact: Upgrade or migration may change expected behavior: v3.48.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.48.0. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.48.0

11. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.48.1 — perf(retrieval): memoize tokenization
• User impact: Upgrade or migration may change expected behavior: v3.48.1 — perf(retrieval): memoize tokenization
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.48.1 — perf(retrieval): memoize tokenization. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.48.1

12. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Developers should check this migration risk before relying on the project: v3.48.2
• User impact: Upgrade or migration may change expected behavior: v3.48.2
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.48.2. Context: Observed during version upgrade or migration.
• Evidence: failure_mode_cluster:github_release | https://github.com/Patdolitse/piia-engram/releases/tag/v3.48.2

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using piia-engram with real data or production workflows.

• [[Bug] 'engram setup' deletes users existing configs without backups.](https://github.com/Patdolitse/piia-engram/issues/23) - github / github_issue
• Community source 2 - github / github_issue
• piia-engram v3.49.0 - github / github_release
• v3.48.2 - github / github_release
• v3.48.1 — perf(retrieval): memoize tokenization - github / github_release
• v3.48.0 - github / github_release
• v3.47.1 - Public Truth Sync - github / github_release
• Engram v3.47.0 - github / github_release
• v3.46.0 - github / github_release
• v3.45.3 - Publication boundary correction - github / github_release
• v3.45.2 - CI entry-point patch - github / github_release
• v3.45.1 - CI packaging patch - github / github_release