Related Pages

Related topics: Installation, Quick Start Guide, System Architecture

Overview

piia-engram (also referred to as Engram) is a local-first, privacy-preserving AI knowledge and memory management system designed as a Model Context Protocol (MCP) server. It enables AI agents and developers to maintain persistent, searchable, and encrypted knowledge corpora that remain entirely under local control.

Source: README.md

Related Pages

Related topics: Quick Start Guide, Setup Wizard

# Installation

This page covers all supported methods for installing and setting up **Engram** (piia-engram), a local-first memory system for AI agents. Engram stores knowledge, decisions, and context in a local corpus with identity-layer encryption and governance controls.

## Overview

Engram supports multiple installation methods targeting different environments and use cases. The project is a Python package distributed via PyPI, with platform-specific shell installers for Linux/macOS and PowerShell installers for Windows. Source: [pyproject.toml:1-15]()

graph TD A[Choose Installation Method] --> B[Package Managers] A --> C[Container] A --> D[Portable Scripts]

B --> B1[pip install piia-engram] B --> B2[uv pip install piia-engram] B --> B3[uv tool install piia-engram]

C --> C1[Docker Build] C --> C2[Docker Pull]

D --> D1[install.sh - Unix/macOS] D --> D2[install.ps1 - Windows]

B1 --> E[Engram CLI: engram] B2 --> E B3 --> E C1 --> F[Docker Container] C2 --> F D1 --> G[Interactive Setup] D2 --> G

## Prerequisites

Before installing Engram, ensure your system meets the following requirements:

| Requirement | Minimum Version | Notes |
|-------------|------------------|-------|
| Python | 3.11+ | Required for core package |
| pip/uv | Latest recommended | uv provides faster installs |
| Git | 2.30+ | For version control integration |
| Disk Space | 500 MB | For package and data storage |

### Optional Dependencies

For full feature availability, the following optional dependencies may be installed:

| Feature | Dependency | Purpose |
|---------|------------|---------|
| Vector Search | sentence-transformers | Semantic embedding for knowledge |
| GUI AI Tools | piia-engram-mcp | MCP server for Claude Desktop, Trae, etc. |
| Docker | docker | Containerized deployment |

## Installation Methods

### Method 1: Package Managers (Recommended)

#### Using pip

The standard Python package installer method:

pip install piia-engram

Source: [pyproject.toml:1-15]()

#### Using uv

The faster Rust-based Python package manager (recommended for development):

Install as a regular dependency

uv pip install piia-engram

Install as a tool (provides console entry points)

uv tool install piia-engram

The `uv tool install` method is preferred because it automatically creates executable entry points for the `engram` CLI and `piia-engram-mcp` server. Source: [pyproject.toml:15-25]()

#### Zero-Install MCP Setup with uvx

For MCP clients that want to use Engram without a permanent installation, use `uvx`:

uvx --from piia-engram piia-engram-mcp

This method pulls the latest version on-demand and is ideal for GUI AI tools like Claude Desktop or Trae. Source: [v3.37.0 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.37.0)

### Method 2: Shell Script Installers

#### Unix/macOS: install.sh

The `install.sh` script provides an interactive installation experience for Unix-like systems:

curl -fsSL https://raw.githubusercontent.com/Patdolitse/piia-engram/main/install.sh | bash

**Features of install.sh:**

- Detects the current operating system and architecture
- Creates necessary directories under `~/.local/share/engram/`
- Sets up shell completion and environment variables
- Verifies installation integrity with checksum validation

Source: [install.sh](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/install.sh)

#### Windows: install.ps1

Windows users can use the PowerShell installer:

irm https://raw.githubusercontent.com/Patdolitse/piia-engram/main/install.ps1 | iex

**Features of install.ps1:**

- Supports Windows PowerShell 5.1+ and PowerShell Core 7+
- Detects UTF-8 encoding issues and configures `PYTHONIOENCODING` if needed
- Creates shortcuts and PATH entries
- Handles Windows-specific directory structures (`%APPDATA%`, `%LOCALAPPDATA%`)

Source: [install.ps1](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/install.ps1)

### Method 3: Docker Container

Engram can be run in a Docker container for isolated execution:

Build from the project's Dockerfile

FROM python:3.11-slim

WORKDIR /app COPY . /app RUN pip install -e .

ENTRYPOINT ["engram"]

**Building and running:**

Build the image

docker build -t engram:latest .

Run with local data volume

docker run -v ~/.local/share/engram:/data engram:latest status

Source: [Dockerfile](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/Dockerfile)

**Docker volume considerations:**

| Volume Path | Purpose | Permissions |
|-------------|---------|-------------|
| `~/.local/share/engram` | Knowledge corpus | Read/Write |
| `/app/logs` | Application logs | Read/Write |

## Post-Installation Setup

After installing Engram, run the initial setup wizard:

engram setup

The setup process performs the following steps:

1. **Corpus initialization** — Creates the local knowledge database
2. **Encryption key generation** — Sets up identity-layer encryption (v3.36.0+)
3. **MCP client configuration** — Detects and configures compatible AI tools
4. **Trust tier assignment** — Establishes governance boundaries

Source: [v3.36.0 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.36.0)

### GUI AI Tool Setup

Engram integrates with several GUI AI tools via the MCP protocol:

| Tool | Config File | Setup Command |
|------|-------------|---------------|
| Claude Desktop | `~/.claude.json` | `engram setup claude` |
| Trae | `~/.trae/mcp.json` | `engram setup trae` |
| Tencent Hunyuan | Platform-specific | `engram setup tencent` |

Source: [v3.37.0 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.37.0)

**Example MCP configuration for Claude Desktop:**

{ "mcpServers": { "engram": { "command": "uvx", "args": ["--from", "piia-engram", "piia-engram-mcp"] } } }

## Verifying Installation

After installation, verify that Engram is correctly installed:

Check CLI availability

engram --version

Run diagnostics

engram doctor

View system status

engram status

The `engram doctor` command checks terminal encoding settings and distinguishes between display issues (mojibake) and actual data corruption. Source: [v3.39.1 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.39.1)

### Expected output from `engram status`

Engram Status ============= Version: 3.40.0 Corpus: ~/.local/share/engram Encryption: Enabled Governance: Active MCP Clients: 2 configured

## Configuration

### Environment Variables

Engram respects the following environment variables:

| Variable | Default | Purpose |
|----------|---------|---------|
| `ENGRAM_ROOT` | `~/.local/share/engram` | Corpus data directory |
| `ENGRAM_EMBED_MODEL` | (built-in default) | Embedding model for vector search |
| `PYTHONIOENCODING` | `utf-8` | Terminal encoding (set automatically) |

### Directory Structure

After installation, Engram creates the following directory structure:

~/.local/share/engram/ ├── corpus/ # Encrypted knowledge storage │ ├── knowledge/ # Knowledge entries │ ├── decisions/ # Decision threads │ └── sessions/ # Saved agent sessions ├── config/ # Configuration files │ ├── trust.json # Trust tier definitions │ └── policy/ # Governance policies ├── logs/ # Application logs └── cache/ # Temporary cache

## Troubleshooting

### Common Installation Issues

| Issue | Cause | Resolution |
|-------|-------|------------|
| `command not found: engram` | PATH not updated | Restart shell or run `source ~/.bashrc` |
| Mojibake in output | Terminal encoding mismatch | Run `engram doctor` to diagnose; set `PYTHONIOENCODING=utf-8` |
| Permission denied | Directory permissions | Check `~/.local/share/engram` permissions |
| MCP connection failed | Config file error | Run `engram setup` to regenerate configs |

### Encoding Diagnostics

If you experience character display issues, the `engram doctor` command provides detailed diagnostics:

engram doctor --encoding

This command:
- Detects terminal encoding capabilities
- Recognizes Windows `cp65001` as UTF-8
- Warns only when actual encoding mismatches exist

Source: [v3.39.1 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.39.1)

### Repairing Corrupted Data

For data corruption issues, use the encoding repair tool:

Dry-run to see what would be repaired

engram repair-encoding --dry-run

Apply repairs with timestamped backup

engram repair-encoding --apply

Source: [v3.38.0 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.38.0)

## Upgrading

### Package Manager Upgrades

pip

pip install --upgrade piia-engram

uv

uv pip install --upgrade piia-engram uv tool upgrade piia-engram

### Reindexing After Upgrades

When upgrading to a new version with embedding model changes, Engram automatically detects the change and triggers reindexing. To manually trigger a reindex:

engram reindex

This ensures vector embeddings are regenerated with the new model, preventing silent degradation of semantic search quality. Source: [v3.33.1 Release Notes](https://github.com/Patdolitse/piia-engram/releases/tag/v3.33.1)

## Uninstallation

### Package Manager Removal

pip

pip uninstall piia-engram

uv

uv tool uninstall piia-engram

### Removing Data

To completely remove Engram and all local data:

Remove the corpus and configuration

rm -rf ~/.local/share/engram

Remove shell completions (manual)

Edit ~/.bashrc or ~/.zshrc and remove engram-related lines

## See Also

- [README.md](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/README.md) — Project overview and quick start
- [Security Architecture](wiki/security-architecture) — Identity-layer encryption and governance
- [MCP Integration](wiki/mcp-integration) — Connecting Engram to AI tools
- [CLI Reference](wiki/cli-reference) — Complete `engram` command documentation

Related Pages

Related topics: Installation, Setup Wizard

Quick Start Guide

Welcome to Engram, a local-first knowledge management system designed for AI agents and developers. This guide walks you through installation, initial setup, and basic usage patterns to get you productive in minutes.

Overview

Engram is a privacy-focused knowledge management tool that runs entirely on your local machine. It provides:

• Knowledge Corpus: Store and retrieve structured knowledge with semantic and keyword search
• MCP Server Integration: Connect AI tools like Claude Desktop, Cursor, and Trae directly to your knowledge base
• Session Continuity: Track cross-tool agent sessions with saved state
• Decision Tracking: Maintain decision threads with provenance and supersession chains
• Encoding Repair: Diagnose and fix text encoding issues automatically

Core Principle: Engram never depends on third-party cloud services. All data remains on your machine.

Source: README.md:1-10

Prerequisites

Before installing Engram, ensure your system meets these requirements:

Required Environment Configuration

Set the following environment variables before first-run:

# Required for terminal encoding (Windows especially)
export PYTHONIOENCODING=utf-8

# Optional: Override default embedding model
# export ENGRAM_EMBED_MODEL=text-embedding-3-small

Source: src/piia_engram/cli.py:42-58

Installation

Method 1: pip (Recommended for CLI Users)

pip install piia-engram

This installs the engram CLI and all core dependencies.

Source: pyproject.toml:1-15

Method 2: uvx (Zero-Install for MCP Clients)

For MCP server integration without full installation:

uvx --from piia-engram piia-engram-mcp

Source: .mcp/server.json:1-20

Method 3: Development Installation

git clone https://github.com/Patdolitse/piia-engram.git
cd piia-engram
pip install -e ".[dev]"

First-Run Setup

The first time you run engram, the setup wizard guides you through configuration.

engram setup

Setup Wizard Flow

graph TD
    A[engram setup] --> B[Detect Environment]
    B --> C{Has existing config?}
    C -->|No| D[Create Engram Root]
    C -->|Yes| E[Load existing config]
    D --> F[Configure MCP Clients]
    E --> F
    F --> G[Initialize Knowledge Corpus]
    G --> H[Run diagnostics]
    H --> I[Setup complete]

MCP Client Configuration

The setup wizard supports configuring multiple MCP clients. Currently supported:

Source: src/piia_engram/setup_wizard.py:85-120

Verifying Installation

After setup, verify everything works correctly:

engram status

The status command provides an MCP client configuration health summary with redacted metadata. For HTML output with detailed tables:

engram status --html

Status Output Interpretation

Source: src/piia_engram/status.py:45-80

Running Diagnostics

If you encounter issues, use the doctor command:

engram doctor

This command:

1. Checks terminal encoding configuration
2. Validates Engram root permissions
3. Tests MCP server connectivity
4. Reports corpus integrity

Source: src/piia_engram/cli.py:120-145

Terminal Encoding Diagnostics

v3.39.1 introduced enhanced encoding diagnostics that distinguish between:

• Terminal display mojibake (visual artifacts)
• Actual data corruption in Engram files

The command recognizes Windows cp65001 as UTF-8 and avoids false warnings when stdout/stderr are already UTF-8.

Source: src/piia_engram/cli.py:130-135

Basic CLI Commands

Knowledge Management

# Add knowledge to your corpus
engram add --type decision --content "Use PostgreSQL for user data"

# Search knowledge base
engram search "database selection"

# List recent entries
engram list --limit 10

Session Management

# List saved agent sessions
engram sessions

# Review staged knowledge
engram review

Source: demos/setup_engram.py:30-55

MCP Server Integration

Quick MCP Setup

Add Engram as an MCP server in your AI tool's configuration:

{
  "mcpServers": {
    "engram": {
      "command": "uvx",
      "args": ["--from", "piia-engram", "piia-engram-mcp"]
    }
  }
}

Source: .mcp/server.json:1-25

GUI Setup via CLI

For GUI tools that support it, use the built-in setup:

engram setup --client claude-desktop

This automatically configures the specified client without manual file editing.

Source: src/piia_engram/setup_wizard.py:150-180

Data Flow Architecture

graph LR
    subgraph "User Layer"
        A[CLI Commands]
        B[MCP Clients]
    end
    
    subgraph "Engram Core"
        C[Knowledge Corpus]
        D[Search Engine]
        E[Decision Tracker]
    end
    
    subgraph "Storage Layer"
        F[SQLite Database]
        G[Vector Index]
        H[File System]
    end
    
    A --> C
    B --> C
    C --> F
    C --> G
    C --> H
    D --> G
    E --> F

Common Workflows

Workflow 1: Daily Knowledge Capture

# Capture a decision
engram add decision "Migrated auth to OAuth2"

# Verify it's indexed
engram search "OAuth2 auth migration"

Workflow 2: AI Agent Session Review

# After an agent session
engram sessions

# Review what was captured
engram review

Workflow 3: Encoding Repair

If you encounter mojibake in terminal output:

# Dry-run to see what would be fixed
engram repair-encoding

# Apply fixes with backup
engram repair-encoding --apply

Source: demos/setup_engram.py:60-85

Configuration Reference

Environment Variables

Directory Structure

~/.engram/
├── corpus/          # Knowledge content (encrypted at rest)
├── index/           # Vector and FTS indexes
├── sessions/        # Saved agent sessions
├── decisions/       # Decision thread data
└── config.json      # Local configuration

Source: src/piia_engram/setup_wizard.py:200-220

Troubleshooting

Issue: Mojibake in Output

This is likely a terminal encoding issue, not data corruption. Run:

engram doctor

The diagnostic will distinguish between display issues and actual corruption.

Source: src/piia_engram/cli.py:130-135

Issue: MCP Client Not Connecting

1. Verify the MCP server is running: engram status
2. Check client config has correct path to piia-engram-mcp
3. Review logs: engram status --html

Issue: Empty Search Results After Adding Content

The index may need rebuilding:

engram reindex

This rebuilds both keyword and vector indexes.

Source: src/piia_engram/cli.py:180-195

Next Steps

After completing this guide:

1. Explore CLI Commands: Run engram --help for all available commands
2. Configure MCP Clients: Connect your preferred AI tools
3. Review Session Features: Use engram sessions to track agent workflows
4. Understand Security: Review the Identity-Layer Security features in v3.36.0

See Also

• README.md - Project overview and detailed documentation
• CLI Reference - Complete command documentation
• MCP Server Setup - Detailed MCP integration guide
• Release Notes - Version history and changelog
• v3.40.0 Release Notes - First-run confidence improvements
• v3.39.1 Release Notes - Terminal encoding diagnostics
• v3.37.0 Release Notes - GUI entry adoption

Related Pages

Related topics: Quick Start Guide, MCP Tools Reference

Related Source Files

The following source files were used to generate this page:

- [src/piia_engram/setup_wizard.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/setup_wizard.py)
- [demos/setup_engram.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/demos/setup_engram.py)
- [src/piia_engram/cli.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/cli.py)
- [src/piia_engram/config.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/config.py)
- [src/piia_engram/mcp_config.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/mcp_config.py)
- [README.md](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/README.md)

# Setup Wizard

The Setup Wizard is Engram's guided initialization system that walks users through configuring their local-first knowledge corpus. It handles the creation of the Engram root directory, establishes the initial knowledge graph structure, configures MCP (Model Context Protocol) client integrations, and applies security settings such as corpus encryption.

## Overview

When a user runs `engram setup`, the wizard performs a series of initialization tasks that establish the foundation for all subsequent Engram operations. The setup process is designed to be idempotent—running it multiple times is safe and will not corrupt existing data.

graph TD A[User runs engram setup] --> B{Engram Root Exists?} B -->|No| C[Create Engram Root Directory] B -->|Yes| D[Verify Existing Configuration] C --> E[Initialize Knowledge Graph DB] D --> E E --> F[Configure MCP Clients] F --> G[Apply Security Settings] G --> H[Display Setup Summary] H --> I[Setup Complete]

## Command Interface

The Setup Wizard is invoked through the `engram setup` CLI command. It accepts several flags to customize the initialization behavior.

### Basic Usage

engram setup

### Command Flags

| Flag | Description | Default |
|------|-------------|---------|
| `--gui` | Enable GUI-based AI tool integration | `false` |
| `--cursor` | Configure for Cursor IDE | `false` |
| `--claude-desktop` | Configure for Claude Desktop | `false` |
| `--trae` | Configure for Trae Editor | `false` |
| `--windsurf` | Configure for Windsurf IDE | `false` |
| `--help` | Display help message | N/A |

Source: [src/piia_engram/cli.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/cli.py)

## MCP Client Configuration

The Setup Wizard supports configuration of multiple MCP clients simultaneously. This allows users to integrate Engram with various AI-assisted development environments.

### Supported Clients

The following table lists the MCP clients that the Setup Wizard can configure:

| Client | Config File | Description |
|--------|-------------|-------------|
| Cursor | `~/.cursor/mcp.json` | AI-first code editor |
| Claude Desktop | `~/.claude_desktop_config.json` | Anthropic's desktop application |
| Trae | `~/.trae/mcp.json` | Tencent's AI code editor |
| Windsurf | `~/.codeium/workspace/mcp_config.json` | Codeium's AI IDE |

Source: [src/piia_engram/mcp_config.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/mcp_config.py)

### Configuration Structure

The MCP configuration uses the standard MCP JSON format. The Setup Wizard generates a configuration entry that points to the `piia-engram-mcp` console script, which can be invoked using `uvx` for zero-install operation:

{ "mcpServers": { "engram": { "command": "uvx", "args": ["--from", "piia-engram", "piia-engram-mcp"] } } }

This zero-install approach (introduced in v3.37.0) eliminates the need for users to manually install the package before configuring MCP clients.

## Initialization Workflow

The Setup Wizard executes a structured initialization sequence that ensures all required components are properly configured.

### Step 1: Engram Root Creation

The wizard first determines the location of the Engram root directory, typically `~/.engram/`. If the directory does not exist, it creates the complete directory structure including subdirectories for the knowledge graph, configuration, and encrypted corpus storage.

### Step 2: Knowledge Graph Initialization

The knowledge graph database is initialized using SQLite with graph extensions. This establishes the schema for nodes, edges, and metadata required for knowledge representation.

### Step 3: Configuration File Setup

The wizard creates the `engram.json` configuration file in the Engram root with sensible defaults:

{ "version": "3.40.0", "corpus_encryption": "enabled", "trust_tier": "local", "embed_model": "default" }

### Step 4: Security Configuration

If corpus encryption is enabled (as it is by default since v3.36.0), the wizard ensures that the encryption layer is properly initialized. Every AI tool sees its own permission boundary inline, and the governance layer is sealed against both write bypass and read-path side effects.

## Demo Integration

A demonstration script is provided in the repository that showcases the setup workflow programmatically.

**File:** [demos/setup_engram.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/demos/setup_engram.py)

This demo illustrates how to:
1. Initialize an Engram instance programmatically
2. Configure MCP clients for specific IDEs
3. Verify the setup was successful

## First-Run Confidence

Since v3.40.0, the Setup Wizard contributes to first-run confidence by ensuring that all configuration is properly established before users begin adding knowledge. The `engram status` command provides a summary of the setup health after initialization.

After running `engram setup`, users are encouraged to run:

engram status

This displays the current configuration state including MCP client configuration health (with redacted metadata) and recommended next steps.

## Troubleshooting

If the Setup Wizard encounters issues, the following diagnostic commands are available:

| Command | Purpose |
|---------|---------|
| `engram doctor` | Run terminal encoding diagnostics and verify setup integrity |
| `engram status` | Display current configuration state and health |
| `engram repair-encoding` | Repair mojibake in Engram data files (dry-run by default) |

### Common Issues

**Issue:** MCP clients not appearing after setup  
**Resolution:** Verify the MCP config file was created in the correct location for your client. Run `engram doctor` to check configuration health.

**Issue:** Unicode/encoding warnings during setup  
**Resolution:** Ensure your terminal encoding is set to UTF-8. The `engram doctor` command can distinguish between terminal display mojibake and actual data corruption (v3.39.1).

## Configuration Options

The Setup Wizard respects several environment variables that influence its behavior:

| Variable | Description | Default |
|----------|-------------|---------|
| `ENGRAM_ROOT` | Override the default Engram root directory | `~/.engram/` |
| `ENGRAM_EMBED_MODEL` | Specify the embedding model for semantic search | System default |
| `PYTHONIOENCODING` | Terminal encoding (recommended: `utf-8`) | Platform default |

## See Also

- [CLI Reference](https://github.com/Patdolitse/piia-engram/wiki/CLI-Reference) - Complete command-line interface documentation
- [MCP Integration](https://github.com/Patdolitse/piia-engram/wiki/MCP-Integration) - Detailed MCP client configuration guide
- [Security Model](https://github.com/Patdolitse/piia-engram/wiki/Security-Model) - Corpus encryption and governance documentation
- [Troubleshooting Guide](https://github.com/Patdolitse/piia-engram/wiki/Troubleshooting) - Common issues and resolutions

Related Pages

Related topics: Core Components, Security and Encryption

System Architecture

Overview

Piia Engram (piia-engram) is a local-first knowledge memory system designed for AI developers and agents. The architecture emphasizes privacy by default, zero external dependencies, and structured knowledge management through the Model Context Protocol (MCP). Source: src/piia_engram/__init__.py:1-20

The core philosophy is that all knowledge content remains local, encrypted at rest, with every AI tool operating within its own permission boundary. This design enables AI assistants and agents to maintain persistent context across sessions while preserving data sovereignty.

High-Level Architecture

graph TD
    subgraph "Client Layer"
        CLI[CLI: engram]
        MCP[MCP Clients]
        GUI[GUI Tools]
    end

    subgraph "Service Layer"
        MCPS[MCP Server]
        Core[Core Engine]
        Search[Search Engine]
        Gov[Governance Layer]
    end

    subgraph "Data Layer"
        Storage[(Storage Backend)]
        Index[(Index Store)]
        Crypto[(Encrypted Corpus)]
        Config[(Config/State)]
    end

    CLI --> Core
    MCP --> MCPS
    MCPS --> Core
    GUI --> MCP

    Core --> Search
    Core --> Gov
    Core --> Storage

    Search --> Index
    Storage --> Crypto
    Gov --> Config

    Storage -.-> Index

Core Components

Piia Engram comprises several interconnected subsystems that work together to provide persistent memory capabilities.

Component Overview

Data Layer Architecture

Storage Subsystem

The storage layer provides the foundation for persistent knowledge management. It implements a file-based storage model with encryption at rest.

graph LR
    A[Knowledge Content] --> B[Encryption Layer]
    B --> C[File System]
    C --> D[Storage Index]
    D --> E[Query Interface]

Key Storage Features:

• Corpus Encryption: All knowledge content is encrypted at rest using identity-layer security. Source: encryption.py
• Freshness Fingerprinting: Storage tracks content changes and embedding model changes to trigger index rebuilds when necessary. Source: storage.py
• Backup System: Encoding repairs create timestamped backups before applying fixes. Source: src/piia_engram/repair.py

Data Models

The system uses structured data models for organizing knowledge:

Source: models.py

Search Architecture

Hybrid Search Implementation

Piia Engram implements a hybrid search strategy combining keyword matching with semantic vector search.

graph TD
    Q[Query] --> KW[Keyword Search]
    Q --> SEM[Semantic Search]

    KW --> RRF[RRF Reranking]
    SEM --> RRF

    RRF --> FINAL[Final Results]

    RRF -.->|Preserve| KWHits[Keyword Hits]

Search Features:

Source: search.py

Index Freshness Management:

• Content changes trigger index updates
• Embedding model changes force complete index rebuild
• Vector backend availability is tracked in freshness fingerprints

Source: storage.py - "freshness fingerprint"

MCP Integration

Model Context Protocol Server

The MCP server provides a standardized interface for AI tools to interact with Engram's memory capabilities.

graph LR
    AI[AI Tool] --> MCP[MCP Client]
    MCP -->|JSON-RPC| Server[MCP Server]
    Server --> Core[Core Engine]
    Core --> Response[Tool Response]
    Response --> AI

MCP Entry Points:

Source: mcp_server.py

Supported GUI Tools:

• Trae (~/.trae/mcp.json)
• Tencent platforms

Source: cli.py - "setup" command

MCP Client Configuration

The system provides diagnostic capabilities for MCP client health:

engram status
engram status --html

The status command summarizes MCP client configuration with redacted metadata, helping users understand their current setup without exposing sensitive information. Source: core.py

Security Layer

Identity-Layer Security

The security architecture implements multiple layers of protection:

graph TD
    subgraph "Security Layers"
        E[Encryption at Rest]
        P[Permission Profiles]
        G[Governance]
        T[Trust Tiers]
    end

    E --> P
    P --> G
    G --> T

Security Components:

Source: v3.36.0 Release Notes

Governance Architecture

The governance layer ensures consistent permission enforcement:

1. Trust-tier checks execute on every read path
2. File-side-effect harness validates all governed operations
3. Playbook policies define allowed operations per trust level

Source: governance.py

CLI Architecture

Command Structure

The CLI provides comprehensive commands for all operations:

Source: cli.py

Status Command Output

The engram status command provides:

• MCP Clients table with redacted metadata
• Next recommended commands (engram doctor, engram review, engram sessions)
• Configuration health summary

Source: v3.40.0 Release Notes

Decision Management

Decision Threads

The system tracks decision evolution through decision threads:

graph LR
    D1[Decision v1] -->|supersedes| D2[Decision v2]
    D2 -->|supersedes| D3[Decision v3]

Features:

• Auto-supersedes: New decisions automatically create supersedes edges
• History tracing: Users can trace how decisions evolved
• Thread continuity: Decision context persists across sessions

Source: v3.35.0 Release Notes

Configuration

Environment Variables

Configuration Files

Source: storage.py

Data Flow: Knowledge Addition

sequenceDiagram
    participant User as CLI/MCP Client
    participant Core as Core Engine
    participant Crypto as Encryption Layer
    participant Storage as Storage Backend
    participant Index as Search Index
    participant Gov as Governance

    User->>Core: add_knowledge(content)
    Core->>Gov: check_permission(tool_id, write)
    Gov-->>Core: allowed
    Core->>Crypto: encrypt(content)
    Crypto-->>Core: encrypted_blob
    Core->>Storage: persist(encrypted_blob)
    Storage->>Index: update_index(content, embedding)
    Core-->>User: success

Common Failure Modes

Encoding Issues

Source: v3.39.1 Release Notes

Index Freshness Issues

Source: v3.33.1 Release Notes

MCP Connection Issues

Source: core.py

Local-First Principles

Piia Engram's architecture strictly follows local-first principles:

1. No cloud dependencies: All data stays on the local filesystem
2. Zero third-party services: No external API calls for core functionality
3. Encryption at rest: Knowledge corpus is always encrypted
4. Permission isolation: Each AI tool sees only its permitted content

This design was explicitly confirmed when declining third-party API partnership proposals, emphasizing that recommending specific API relay services contradicts the local-first positioning. Source: Issue #8

See Also

• Installation Guide — Setting up piia-engram locally
• CLI Reference — Complete command documentation
• MCP Integration — Connecting AI tools to Engram
• Security Model — Deep dive into encryption and governance
• Search Configuration — Tuning hybrid search performance

Related Pages

Related topics: System Architecture, MCP Tools Reference, Security and Encryption

Core Components

Piia Engram is a local-first knowledge management system designed for AI agents and developers who need persistent, privacy-preserving memory capabilities. The system provides encrypted storage, governance-based access control, and Model Context Protocol (MCP) integration for seamless AI tool connectivity.

System Architecture Overview

The architecture follows a layered design with clear separation of concerns between storage, security, governance, and interface layers.

graph TD
    subgraph "Interface Layer"
        CLI[CLI: engram]
        MCP[MCP Server]
        API[Python API]
    end
    
    subgraph "Governance Layer"
        GR[Governance Runtime]
        GOV[Governance Engine]
        GCONF[Governance Config]
    end
    
    subgraph "Core Layer"
        CORE[Core Engine]
        HOOKS[Hooks System]
        SEARCH[Search Engine]
    end
    
    subgraph "Security Layer"
        CRYPTO[Encryption]
        AUTH[Trust Tiers]
    end
    
    subgraph "Storage Layer"
        KB[Knowledge Base]
        SESSIONS[Session Store]
        IDX[Index Store]
    end
    
    CLI --> CORE
    MCP --> CORE
    API --> CORE
    
    CORE --> GR
    CORE --> HOOKS
    CORE --> SEARCH
    
    CORE --> CRYPTO
    GR --> AUTH
    
    CORE --> KB
    CORE --> SESSIONS
    SEARCH --> IDX
    
    KB --> CRYPTO
    SESSIONS --> CRYPTO
    
    GR --> GOV
    GOV --> GCONF

Source: src/piia_engram/core.py:1-50, src/piia_engram/governance_runtime.py:1-30

Related Pages

Related topics: Knowledge Management, Search and Retrieval, Core Components

MCP Tools Reference

This page documents the MCP (Model Context Protocol) tools exposed by piia-engram, enabling AI agents and GUI clients to interact with Engram's knowledge corpus, decision tracking, and session management capabilities.

Overview

piia-engram exposes a comprehensive set of MCP tools that allow AI agents to persistently store, retrieve, and manage knowledge artifacts. The MCP server acts as the interface layer between AI clients and Engram's local-first knowledge infrastructure. Source: src/piia_engram/mcp_server.py:1-50

graph TD
    A["MCP Client<br/>(Claude Desktop, Cursor, Trae, etc.)"] --> B["piia-engram-mcp<br/>Server"]
    B --> C["Knowledge Corpus<br/>SQLite + FTS + Vector"]
    B --> D["Decision Graph<br/>(edges & vertices)"]
    B --> E["Session Store<br/>(JSONL)"]
    B --> F["Governance Layer<br/>(trust-tier enforcement)"]
    
    G["CLI: engram doctor"] --> B
    H["CLI: engram status"] --> B
    I["CLI: engram sessions"] --> B

Tool Categories

MCP Server Architecture

The MCP server is implemented as a Python package with a console entry point. Source: src/piia_engram/mcp/__init__.py:1-30

Entry Points

Source: pyproject.toml:1-50 (console_scripts configuration)

Zero-Install MCP Configuration

For GUI-based AI tools that support MCP, the recommended setup uses uvx for zero-installation:

{
  "mcpServers": {
    "engram": {
      "command": "uvx",
      "args": ["--from", "piia-engram", "piia-engram-mcp"]
    }
  }
}

This configuration is automatically generated by engram setup for supported GUI clients. Source: v3.37.0 Release Notes

Supported GUI Clients

Tool Reference

Knowledge Tools

#### add_knowledge

Stores new knowledge artifacts in the Engram corpus.

Returns: Knowledge record ID and confirmation.

Example Request:

{
  "tool": "add_knowledge",
  "parameters": {
    "content": "The authentication service uses JWT tokens with 1-hour expiry",
    "metadata": {
      "source": "api-docs",
      "component": "auth"
    }
  }
}

Source: src/piia_engram/mcp_server.py:100-150

#### search_knowledge

Performs hybrid search combining keyword and semantic retrieval.

Hybrid Search Behavior:

• Keyword results are preserved and ranked using RRF (Reciprocal Rank Fusion)
• Semantic (vector) results supplement keyword matches
• Ensures hybrid recall ≥ keyword-only recall

Source: v3.33.2 Release Notes

Decision Tools

#### add_decision

Records a decision in the decision graph with automatic threading.

Auto-supersedes Behavior: When the same question receives a new answer, add_decision automatically creates a supersedes edge to the previous decision. This enables tracing decision evolution over time. Source: v3.35.0 Release Notes

#### get_decisions

Retrieves decision history, optionally filtered by question or time range.

Session Tools

#### sessions_list

Lists saved cross-tool agent sessions.

Sessions are stored as JSONL files in the Engram root directory, enabling AI agents to resume previous contexts. Source: v3.39.0 Release Notes

CLI Diagnostic Tools

While MCP tools target AI agents, the CLI provides human-accessible diagnostics:

engram status

Reports MCP client configuration health with redacted metadata.

engram status
engram status --html

The --html output includes:

• MCP Clients table with configuration status
• Next Commands suggestions: engram doctor, engram review, engram sessions

Source: v3.40.0 Release Notes

engram doctor

Performs terminal encoding diagnostics and health checks.

engram doctor

Diagnostics Performed:

• UTF-8 encoding verification (Windows cp65001 recognized)
• Mojibake detection distinguishes display issues from data corruption
• Warning suppression when stdout/stderr are already UTF-8

Source: v3.39.1 Release Notes

engram repair-encoding

Repairs detected mojibake in knowledge files.

engram repair-encoding        # Dry-run mode
engram repair-encoding --apply # Apply repairs with backup

Repair Guardrails:

• Reversible cases are repaired with timestamped backups
• Lossy/suspicious cases are reported for manual review
• Scans JSON, JSONL, Markdown, and text files

Source: v3.38.0 Release Notes

Configuration

Environment Variables

Source: src/piia_engram/cli.py:50-100

Trust Tier Configuration

The governance layer enforces permission profiles based on trust tiers:

graph LR
    A["Local CLI"] --> B["R20: Full Access"]
    C["MCP Client"] --> D["R15: Controlled Access"]
    E["External Export"] --> F["R10: Read-Only"]
    
    B --> G["Corpus Encryption"]
    D --> G
    F --> G

Source: v3.34.0 Release Notes

Security Model

Corpus Encryption

Knowledge content is encrypted at rest. Each AI tool sees its own permission boundary inline. Source: v3.36.0 Release Notes

Governance Closure

The governance layer is sealed against:

• Write bypass attempts
• Read-path side effects

All governed/export tools enforce trust-tier checks on every read path. Source: v3.34.0 Release Notes

Common Failure Modes

Encoding Repair Workflow

graph TD
    A["engram repair-encoding"] --> B{Dry-run or Apply?}
    B -->|Dry-run| C["List suspicious files"]
    B -->|Apply| D["Create timestamped backup"]
    D --> E{Repairable?}
    E -->|Yes| F["Apply repair"]
    E -->|No| G["Report for manual review"]
    C --> H["Review list"]
    F --> I["Verify restored content"]

Data Flow

sequenceDiagram
    participant Client as MCP Client
    participant Server as piia-engram-mcp
    participant Storage as SQLite/JSONL
    participant Vector as Embedding Engine
    
    Client->>Server: add_knowledge(content)
    Server->>Vector: generate_embedding(content)
    Server->>Storage: Store with embedding
    Server-->>Client: record_id
    
    Client->>Server: search_knowledge(query, mode=hybrid)
    Server->>Vector: generate_embedding(query)
    Server->>Storage: FTS + KNN query
    Storage-->>Server: ranked_results
    Server-->>Client: results

See Also

• Home — Project overview and quick start
• CLI Reference — Full CLI command documentation
• Architecture Overview — System architecture details
• Governance & Security — Trust tiers and encryption
• Troubleshooting — Common issues and solutions

Related Pages

Related topics: MCP Tools Reference, Search and Retrieval

Knowledge Management

Overview

Knowledge Management in piia-engram is the core system for capturing, storing, organizing, and retrieving information that AI agents and users need to maintain context across sessions. As a local-first, privacy-focused knowledge management system, Engram provides a structured corpus where decisions, session histories, and absorbed knowledge are stored with encryption at rest and fine-grained access controls.

The knowledge management system is designed to support cross-tool agent workflows, enabling AI assistants to remember previous interactions, decisions made, and context gathered over time without relying on external cloud services. Source: src/piia_engram/corpus.py:1-50

Architecture Overview

The knowledge management architecture consists of several interconnected components that work together to provide a comprehensive memory system for AI agents.

graph TD
    A[User / AI Agent] -->|add_decision| B[Decision Thread Manager]
    A -->|create_session| C[Session Manager]
    A -->|absorb_knowledge| D[Auto Absorb Hook]
    
    B -->|stores| E[(Decision Corpus)]
    C -->|stores| F[(Session Storage)]
    D -->|compacts| G[(Knowledge Corpus)]
    
    H[Search Engine] -->|queries| E
    H -->|queries| G
    H -->|hybrid search| F
    
    I[Governance Layer] -->|enforces permissions| E
    I -->|enforces permissions| G
    I -->|enforces permissions| F
    
    J[Corpus Encryption] -->|encrypts at rest| E
    J -->|encrypts at rest| G
    J -->|encrypts at rest| F

Core Components

Source: src/piia_engram/decision_thread.py:1-100, src/piia_engram/sessions.py:1-80

Decision Threads

Decision threads are the primary mechanism for tracking how decisions evolved over time within the knowledge management system. Each decision records not only the decision itself but also the context, rationale, and relationships to previous decisions.

Decision Thread Structure

classDiagram
    class Decision {
        +str id
        +str question
        +str answer
        +str rationale
        +datetime timestamp
        +list~str~ supersedes
        +str thread_id
        +dict metadata
    }
    
    class DecisionThread {
        +str thread_id
        +str topic
        +list~Decision~ decisions
        +datetime created_at
        +datetime updated_at
    }
    
    DecisionThread "1" --> "*" Decision : contains
    Decision "1" --> "*" Decision : supersedes

Auto-Supersession Behavior

When a new decision is added that addresses the same question as an existing decision, the system automatically creates a supersedes edge. This ensures that decision history remains traceable while making the most recent decision the authoritative one. Source: src/piia_engram/decision_thread.py:45-80

Decision Attributes

Session Management

Sessions represent cross-tool agent workflows that span multiple interactions. The session management system provides continuity by storing session state and enabling resumption of interrupted workflows.

Session Lifecycle

stateDiagram-v2
    [*] --> Active: create_session
    Active --> Staged: auto_save_on_stop hook
    Active --> Suspended: timeout or manual suspend
    Suspended --> Active: resume_session
    Staged --> Archived: review_complete
    Staged --> Active: resume_session
    Archived --> [*]: session_prune

Session Storage

Sessions are stored with the following structure:

Source: src/piia_engram/sessions.py:20-60

Knowledge Lifecycle Hooks

Engram provides automated hooks that manage the knowledge lifecycle without requiring manual intervention.

Auto-Absorb and Compact

The auto-absorb hook captures knowledge from agent interactions and automatically compacts redundant entries to maintain an efficient corpus. Source: src/piia_engram/hooks/auto_absorb_compact.py:1-100

Auto-Inject Resume Brief

When resuming a session, this hook automatically injects relevant context from the knowledge corpus to help the agent regain context quickly. Source: src/piia_engram/hooks/auto_inject_resume_brief.py:1-50

Auto-Save on Stop

This hook ensures session state is persisted when an agent session ends, preventing data loss from unexpected interruptions. Source: src/piia_engram/hooks/auto_save_on_stop.py:1-60

Hook Execution Flow

sequenceDiagram
    participant Agent
    participant HookManager
    participant KnowledgeCorpus
    participant SessionStorage
    
    Agent->>HookManager: Session ends
    HookManager->>HookManager: Trigger auto_save_on_stop
    HookManager->>SessionStorage: Persist session state
    
    HookManager->>HookManager: Trigger auto_absorb_compact
    HookManager->>KnowledgeCorpus: Compact redundant entries
    
    HookManager->>HookManager: Update freshness fingerprint
    KnowledgeCorpus-->>HookManager: Index updated

Knowledge Retrieval

Hybrid Search

Engram supports hybrid search combining full-text search (FTS) with semantic vector search to maximize recall. The system uses Reciprocal Rank Fusion (RRF) to merge result sets while ensuring keyword-matched results are never dropped. Source: src/piia_engram/search.py:1-120

Search Configuration

Index Freshness

The system maintains a freshness fingerprint that tracks:

• Content modifications
• Embedding model changes
• Vector backend availability

When the embedding model changes, the index is automatically rebuilt to ensure vector dimensions match the new model. Source: src/piia_engram/reconcile.py:30-80

Security and Governance

Corpus Encryption

All knowledge content is encrypted at rest. The encryption system ensures that each AI tool sees only its own permission boundary inline, maintaining data isolation between different agents and users. Source: src/piia_engram/corpus.py:100-150

Trust-Tier Enforcement

The governance layer enforces trust-tier checks on every read and write path, closing file-side-effect leak classes and preventing unauthorized access to knowledge entries.

graph LR
    A[Read Request] --> B{Trust Tier Check}
    B -->|Passed| C[Return Knowledge Entry]
    B -->|Failed| D[Access Denied]
    
    E[Write Request] --> F{Governance Check}
    F -->|Passed| G[Persist to Corpus]
    F -->|Failed| H[Write Blocked]

Permission Profile

Users can control who accesses their Engram data through permission profiles that define trust tiers for different agents and tools.

CLI Commands for Knowledge Management

Status Command

The engram status command provides a comprehensive overview of the knowledge management system:

engram status          # Show basic status
engram status --html   # Generate HTML report with MCP clients table

Source: src/piia_engram/status.py:1-100

Doctor Command

The engram doctor command runs diagnostics including terminal encoding checks:

engram doctor          # Run full diagnostic suite

Source: src/piia_engram/doctor.py:1-80

Sessions Command

The engram sessions command lists saved cross-tool agent sessions:

engram sessions        # List all sessions
engram sessions --active  # Show only active sessions

Source: src/piia_engram/sessions.py:80-120

Repair Encoding Command

For repairing mojibake in knowledge files:

engram repair-encoding         # Dry-run for encoding issues
engram repair-encoding --apply # Apply repairs with backup

Common Failure Modes

Data Storage Structure

~/.engram/
├── corpus/              # Knowledge entries with embeddings
├── decisions/           # Decision threads and history
├── sessions/            # Cross-tool agent sessions
├── search_index/        # FTS and vector search indexes
├── governance/          # Permission profiles and policies
└── config.yaml          # Local configuration

See Also

• MCP Setup Guide - Configuring Engram as an MCP server
• CLI Reference - Complete CLI command documentation
• Security Architecture - Encryption and governance details
• Release Notes - Latest features and fixes

Related Pages

Related topics: MCP Tools Reference, Knowledge Management

Related Source Files

The following source files were used to generate this page:

- [src/piia_engram/search_index.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/search_index.py)
- [src/piia_engram/retrieval.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/retrieval.py)
- [src/piia_engram/embeddings.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/embeddings.py)
- [src/piia_engram/config.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/config.py)
- [src/piia_engram/corpus.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/corpus.py)
- [src/piia_engram/database.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/database.py)
- [src/piia_engram/cli.py](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/src/piia_engram/cli.py)
- [docs/hybrid-search.md](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/docs/hybrid-search.md)
- [docs/embeddings.md](https://github.com/Patdolitse/piia-engram/blob/0753eaaf7e8bb4d6dc7e2f3a84415e098903522b/docs/embeddings.md)

Search and Retrieval

This page documents the search and retrieval system in Engram, covering architecture, configuration, usage patterns, and implementation details for knowledge content discovery.

Overview

Engram provides a multi-modal search and retrieval system that enables users and AI agents to locate relevant knowledge content across the local corpus. The system supports three primary search strategies:

1. Keyword Search (FTS) — Exact term matching using SQLite Full-Text Search (FTS5)
2. Semantic Search — Vector-based similarity search using embedding models
3. Hybrid Search — Combines keyword and semantic results using Reciprocal Rank Fusion (RRF)

The search system is designed for local-first operation, processing all data on-device without external API dependencies. Source: src/piia_engram/search_index.py:1-50

Architecture

graph TD
    A[User Query] --> B[Search Entry Point]
    B --> C{Search Mode}
    C -->|keyword| D[FTS5 Query Engine]
    C -->|semantic| E[Embedding Model]
    C -->|hybrid| F[Hybrid Orchestrator]
    
    E --> G[Vector Similarity Search]
    D --> H[Ranked Results]
    F --> I[RRF Fusion]
    G --> I
    
    I --> J[Post-Processing]
    H --> J
    J --> K[Filtered Results]
    K --> L[Trust-Tier Gating]
    L --> M[Final Results]
    
    N[(Corpus Database)] -.-> D
    O[(Vector Index)] -.-> G
    P[Embedding Cache] -.-> E

Core Components

Source: src/piia_engram/search_index.py:50-150

Search Modes

Keyword Search (FTS5)

Full-Text Search provides exact term matching using SQLite's FTS5 extension. This mode is optimal for:

• Known exact terms or phrases
• Code identifiers and technical keywords
• Users who prefer deterministic results

The FTS index supports prefix matching (term*), phrase matching ("exact phrase"), and boolean operators (AND, OR, NOT). Source: src/piia_engram/database.py:200-280

Configuration Options:

Semantic Search

Semantic search converts queries and content into dense vector embeddings, enabling similarity-based retrieval. This mode excels at:

• Conceptual and semantic matches
• Paraphrased queries
• Finding related content without exact keyword overlap

Source: src/piia_engram/embeddings.py:1-100

Embedding Configuration:

graph LR
    A[Query Text] --> B[Embedding Service]
    B --> C[Query Vector]
    C --> D[KNN Search]
    D --> E[Content Vectors]
    E --> F[Top-K Results]
    F --> G[Score Threshold]
    G --> H[Semantic Matches]

Important Index Freshness Behavior:

When the embedding model is changed via ENGRAM_EMBED_MODEL, the vector index is automatically flagged for rebuild. The freshness fingerprint includes the model identifier, ensuring that changing models triggers index reconstruction. Source: src/piia_engram/search_index.py:300-350

Hybrid Search

Hybrid search combines keyword and semantic results using Reciprocal Rank Fusion (RRF), providing both precision and recall. The algorithm:

1. Executes FTS query and semantic query in parallel
2. Normalizes scores from each result set
3. Applies RRF formula: score = Σ(1 / (k + rank)) where k=60
4. Returns merged, re-ranked results

Hybrid Recall Guarantee:

Engram's hybrid search ensures that keyword matches are never lost during fusion. The implementation preserves all FTS results before applying RRF, addressing a known issue where RRF truncation could displace keyword hits. Source: docs/hybrid-search.md:1-50

# Hybrid search result fusion (simplified)
def rrf_fusion(fst_results, semantic_results, k=60):
    fused_scores = {}
    
    # Preserve all keyword results
    for rank, (chunk_id, fts_score) in enumerate(fst_results):
        fused_scores[chunk_id] = fts_score
    
    # Apply RRF for semantic results
    for rank, (chunk_id, vec_score) in enumerate(semantic_results):
        rrf_score = 1 / (k + rank)
        if chunk_id in fused_scores:
            fused_scores[chunk_id] += vec_score + rrf_score
        else:
            fused_scores[chunk_id] = rrf_score
    
    return sorted(fused_scores.items(), key=lambda x: -x[1])

Search Configuration

Environment Variables

Source: src/piia_engram/config.py:1-100

Vector Backend Selection

The system auto-selects the optimal vector backend based on available dependencies:

Source: src/piia_engram/embeddings.py:100-150

Usage Patterns

CLI Search Commands

# Basic keyword search
engram search "exact phrase"

# Semantic search
engram search --mode semantic "conceptual query"

# Hybrid search (default)
engram search --mode hybrid "combined query"

# Limit results
engram search --limit 10 "query"

# Filter by metadata
engram search --source "github" --tag "bug" "query"

Source: src/piia_engram/cli.py:200-300

Python API

from piia_engram import SearchIndex, RetrievalEngine

# Initialize search index
index = SearchIndex(corpus_path="./engram_data")

# Execute hybrid search
results = index.search(
    query="how to configure MCP",
    mode="hybrid",
    limit=10,
    trust_tier="user"
)

for result in results:
    print(f"{result.chunk_id}: {result.score:.3f}")
    print(f"  Content: {result.content[:100]}...")

Source: src/piia_engram/search_index.py:400-450

MCP Tool Integration

The MCP server exposes search functionality for AI agents:

{
  "name": "engram_search",
  "description": "Search Engram knowledge base",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "mode": {"enum": ["keyword", "semantic", "hybrid"]},
      "limit": {"type": "integer", "default": 10}
    }
  }
}

Source: src/piia_engram/cli.py:50-100

Index Management

Index States

graph TD
    A[New Corpus] --> B[Initial Index Build]
    B --> C[Index Ready]
    C --> D{Content Change}
    C --> E{Model Change}
    D --> F[Incremental Update]
    E --> G[Full Rebuild]
    F --> C
    G --> C

Rebuild Commands

# Full index rebuild
engram reindex

# FTS-only rebuild
engram reindex --type fts

# Vector-only rebuild
engram reindex --type vector

# Force rebuild regardless of freshness
engram reindex --force

Vector Backend Availability Detection:

When vector dependencies are installed after FTS-only indexing, the system detects missing vector tables and automatically triggers a rebuild to include semantic search capabilities. Source: src/piia_engram/search_index.py:250-300

Freshness Fingerprint

The index freshness is determined by a fingerprint that includes:

Source: src/piia_engram/corpus.py:150-200

Trust-Tier Gating

All search results pass through trust-tier enforcement, ensuring that AI tools access only permitted content based on their permission profile. Source: src/piia_engram/retrieval.py:100-150

graph TD
    A[Search Results] --> B{Trust-Tier Check}
    B -->|chunk.trust_tier <= tool.trust_tier| C[Include in Results]
    B -->|chunk.trust_tier > tool.trust_tier| D[Exclude]
    C --> E[Post-Processing]
    D --> F[Audit Log Entry]

Common Issues and Solutions

Symptom: Empty Semantic Results After Model Change

Cause: The vector index was built with a different embedding model, causing dimension mismatch.

Solution: The system automatically detects model changes via freshness fingerprint. If results are empty after a model change, run:

engram reindex --type vector --force

This ensures the vector table is rebuilt with the new model's dimensions. Source: docs/hybrid-search.md:80-100

Symptom: Hybrid Search Missing Expected Keyword Matches

Cause: RRF fusion with strict result limits could displace keyword matches.

Solution: Engram v3.33.2+ preserves all keyword results before applying RRF fusion. If using an older version, upgrade or run:

engram reindex

Symptom: Slow Search on Large Corpus

Diagnostic: Check vector backend selection:

engram doctor

Solutions:

1. Enable GPU acceleration: ENGRAM_EMBED_DEVICE=cuda
2. Increase batch size: ENGRAM_EMBED_BATCH_SIZE=64
3. Limit search scope with filters: --source, --tag, --date-range

Symptom: Encoding Issues in Search Results

The engram doctor command includes terminal encoding diagnostics to distinguish between display mojibake and actual data corruption. Source: src/piia_engram/cli.py:300-350

# Run encoding diagnostics
engram doctor --encoding

Performance Considerations

Optimization Tips:

1. Batch operations — When adding multiple chunks, use batch insert for better throughput
2. Cache embeddings — Repeated queries benefit from embedding cache
3. Filter early — Apply metadata filters before search to reduce result set
4. Warm-up queries — First semantic query may be slower due to model loading

Related Documentation

• Hybrid Search Implementation — Detailed RRF algorithm documentation
• Embedding Configuration — Model selection and optimization
• Trust and Governance — Access control and permission profiles
• CLI Reference — Complete command documentation

See Also

• Home — Project overview
• Architecture — System design overview
• MCP Integration — MCP server setup and tools
• Configuration Reference — Full environment variable documentation

Related Pages

Related topics: System Architecture

Security and Encryption

Overview

Engram implements a comprehensive identity-layer security system designed to protect knowledge content at rest while maintaining clear permission boundaries between AI tools and users. The security architecture is built around three core principles: encryption at rest, trust-tier enforcement, and governance isolation.

The system ensures that every AI tool sees only its own permission boundary inline, and the governance layer is sealed against both write bypass and read-path side effects. This approach aligns with Engram's local-first privacy model, where data remains under user control without dependence on third-party cloud services.

Source: SECURITY.md:1-20

Architecture

High-Level Security Model

The security architecture consists of layered components that work together to provide defense-in-depth protection for knowledge content:

graph TD
    A[User Data] --> B[Encryption Layer<br/>crypto.py]
    B --> C[Permission Layer<br/>permission.py]
    C --> D[Governance Layer<br/>governance.py]
    D --> E[Trust Tier Engine<br/>trust.py]
    
    F[AI Tool Request] --> C
    G[Export Request] --> D
    H[Read Request] --> E
    
    style B fill:#e1f5fe
    style C fill:#e8f5e8
    style D fill:#fff3e0
    style E fill:#fce4ec

Encryption at Rest

Knowledge content stored in the Engram corpus is encrypted at rest using cryptographic primitives defined in the crypto module. This ensures that even if storage media are compromised, the content remains protected.

Source: src/piia_engram/crypto.py:1-50

Trust-Tier System

The trust-tier system enforces access controls on every read path. All governed and export tools check trust-tier levels before granting access to knowledge content, closing the file-side-effect leak class that was identified in earlier versions.

Source: src/piia_engram/trust.py:1-30

Core Components

Source: src/piia_engram/governance.py:1-40

Permission Profile

Engram v3.35.0 introduced permission profiles that give users control over who accesses their Engram data. Each AI tool receives its own permission boundary, preventing cross-tool data leakage.

Permission Boundary Enforcement

Every AI tool sees only its own permission boundary inline. This isolation ensures that:

1. Tools cannot read content outside their authorized scope
2. Write operations are restricted to permitted content areas
3. Access attempts are logged for audit purposes

Source: src/piia_engram/permission.py:1-35

Governance Layer

The governance layer acts as the final enforcement point for security policies. It was hardened in v3.34.0 and v3.36.0 to seal against both write bypass and read-path side effects.

graph LR
    A[Request] --> B{Governance Check}
    B -->|Write| C[Write Path Guard]
    B -->|Read| D[Trust Tier Check]
    C -->|Bypass Attempt| E[Blocked]
    D -->|Insufficient Trust| E
    D -->|Pass| F[Content Access]

Governance Sealing

The governance layer implements the following protections:

• Write Bypass Prevention: All write operations must pass through governance validation
• Read-Path Side Effect Prevention: Read operations cannot trigger unintended state changes
• Trust-Tier Enforcement: Every read path enforces trust-tier checks

Source: src/piia_engram/governance.py:40-80

Sensitivity Classification

Content sensitivity is managed through the sensitivity module, which classifies knowledge content and applies appropriate protection levels.

Sensitivity Levels

Source: src/piia_engram/sensitivity.py:1-45

Configuration Options

Environment Variables for Security

Source: src/piia_engram/governance.py:15-25

Usage Patterns

Initializing Secure Storage

from piia_engram import crypto, governance

# Initialize encryption with provided key
crypto.initialize(key_path="/path/to/key")

# Verify governance layer is sealed
governance.verify_integrity()

Trust-Tier Checking

from piia_engram import trust, permission

# Check if a tool has sufficient trust tier for read access
tool_trust = trust.get_tool_trust_level("mcp-client-name")
required_tier = "internal"

if trust.check_access(tool_trust, required_tier):
    # Grant read access
    pass
else:
    # Deny access and log attempt
    trust.log_denial("mcp-client-name", "read", required_tier)

Source: src/piia_engram/trust.py:50-80

Common Failure Modes

Encryption Key Issues

• Missing Key: Engram will refuse to decrypt corpus without valid key
• Corrupted Key: Key validation fails; re-initialization may be required
• Key Rotation: Unrotated keys may cause access issues after policy updates

Trust-Tier Violations

• Insufficient Trust: Tools with low trust tiers cannot access protected content
• Trust Tier Drift: Tools may accumulate permissions over time (use engram status to audit)

Governance Blockages

• Write Bypass Attempt: Blocked and logged; prevents unauthorized content modification
• Read-Path Side Effects: Detected and prevented by governance layer

Source: SECURITY.md:30-60

Security Best Practices

1. Key Management: Store encryption keys securely using a secrets manager
2. Trust Tier Auditing: Regularly audit tool trust tiers using engram status
3. Governance Verification: Periodically run engram doctor to verify governance integrity
4. Audit Log Review: Review audit logs for suspicious access patterns
5. Principle of Least Privilege: Assign minimum necessary trust tiers to tools

Source: SECURITY.md:60-90

See Also

• CLI Reference - Security-related commands
• Installation Guide - Secure installation procedures
• Architecture Overview - System architecture details
• Governance Configuration - Governance layer setup
• Release Notes v3.36.0 - Identity-layer security release
• Release Notes v3.34.0 - Governance cutover details

Doramagic Pitfall Log

Found 18 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.34.0 — Governance cutover + Universal harness + Playbook policy
• User impact: Upgrade or migration may change expected behavior: v3.34.0 — Governance cutover + Universal harness + Playbook policy
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.34.0 — Governance cutover + Universal harness + Playbook policy. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | fmev_76925401e068c1b5f3b688fe03df685f | https://github.com/Patdolitse/piia-engram/releases/tag/v3.34.0

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.35.0 — Decision Threads & Permission Profile
• User impact: Upgrade or migration may change expected behavior: v3.35.0 — Decision Threads & Permission Profile
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.35.0 — Decision Threads & Permission Profile. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | fmev_77e4f97aad82e83fe9ef3312294eec6b | https://github.com/Patdolitse/piia-engram/releases/tag/v3.35.0

3. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.37.0 - GUI entry adoption: piia-engram-mcp + uvx
• User impact: Upgrade or migration may change expected behavior: v3.37.0 - GUI entry adoption: piia-engram-mcp + uvx
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.37.0 - GUI entry adoption: piia-engram-mcp + uvx. Context: Observed during installation or first-run setup.
• Evidence: failure_mode_cluster:github_release | fmev_75eede3829cc5d9e70aae6672a96605c | https://github.com/Patdolitse/piia-engram/releases/tag/v3.37.0

4. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.39.0 - Local Workflow Visibility
• User impact: Upgrade or migration may change expected behavior: v3.39.0 - Local Workflow Visibility
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.39.0 - Local Workflow Visibility. Context: Observed during installation or first-run setup.
• Evidence: failure_mode_cluster:github_release | fmev_500227db57a8529c37cfce701ae84bdc | https://github.com/Patdolitse/piia-engram/releases/tag/v3.39.0

5. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

6. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: v3.38.0 - Encoding repair guardrails
• User impact: Upgrade or migration may change expected behavior: v3.38.0 - Encoding repair guardrails
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.38.0 - Encoding repair guardrails. Context: Observed when using windows
• Evidence: failure_mode_cluster:github_release | fmev_42be567d8e4b76432faa58f51baccf59 | https://github.com/Patdolitse/piia-engram/releases/tag/v3.38.0

7. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: v3.40.0 - First-run confidence
• User impact: Upgrade or migration may change expected behavior: v3.40.0 - First-run confidence
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.40.0 - First-run confidence. Context: Observed during installation or first-run setup.
• Evidence: failure_mode_cluster:github_release | fmev_e8bb2627f41868837723a394026965a3 | https://github.com/Patdolitse/piia-engram/releases/tag/v3.40.0

8. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

9. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

11. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | github_repo:1242620513 | https://github.com/Patdolitse/piia-engram

12. Capability evidence risk: Capability evidence risk requires verification

• Severity: low
• Finding: Project evidence flags a capability evidence risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: failure_mode_cluster:github_issue | fmev_2c63fe4051ac808c161d261e8e1fd5e9 | https://github.com/Patdolitse/piia-engram/issues/8

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using piia-engram with real data or production workflows.

• Community source 1 - github / github_issue
• v3.40.0 - First-run confidence - github / github_release
• v3.39.1 - Terminal Encoding Diagnostics - github / github_release
• v3.39.0 - Local Workflow Visibility - github / github_release
• v3.38.0 - Encoding repair guardrails - github / github_release
• v3.37.0 - GUI entry adoption: piia-engram-mcp + uvx - github / github_release
• v3.36.0 — Identity-layer security: corpus encryption + governance closur - github / github_release
• v3.35.0 — Decision Threads & Permission Profile - github / github_release
• v3.34.0 — Governance cutover + Universal harness + Playbook policy - github / github_release
• Community source 10 - github / github_release
• Community source 11 - github / github_release
• Configuration risk requires verification - GitHub / issue