Related Pages

Related topics: Lib

Related Source Files

The following source files were used to generate this page:

• README.md
• pyproject.toml
• lib/cmd_rebuild.py
• lib/cmd_search.py
• lib/sources/registry.py
• lib/sources/goose.py
• lib/sources/grok.py
• lib/db.py
• lib/embeddings.py
• lib/llm.py

Overview

Total Recall is a Python CLI utility that aggregates AI assistant chat histories from a growing collection of local CLI clients (Claude Code, Codex, Goose, Grok, Gemini, Aider, Cursor, Continue, Cline, and others) into a single, searchable local archive. It exists so that prompts, answers, and tool-call transcripts that live in vendor-specific JSONL or SQLite files on a developer's machine can be preserved, deduplicated, queried, and optionally refined by an LLM — independent of any cloud service. Source: README.md:1-30

The project positions itself as a personal "black box" for AI interactions: even when a CLI client changes its on-disk format, gets uninstalled, or rotates session IDs, the rebuild command re-reads the raw source files and re-populates the local archive. The CLI entry point is total-recall, declared in pyproject.toml:30-45 under the [project.scripts] table.

Architecture at a Glance

The codebase is organised into a small command layer and a plug-in style source registry.

Each source adapter inherits from a common Source base class and implements iter_sessions() returning normalised (source_name, session_id, ts, role, content, cwd, model) tuples. The registry in lib/sources/registry.py:10-60 aggregates all enabled adapters and is the single point consulted by the rebuild command. Source: lib/sources/registry.py:15-45

Core Workflow

The lifecycle of an entry in the archive is straightforward:

1. Discover — cmd_rebuild.py walks the registry, calling iter_sessions() on every adapter. Adapters like the Goose adapter open ~/.local/share/goose/sessions/sessions.db and stream rows from SQLite; the Grok adapter reads JSONL files under ~/.grok/sessions/ and URL-decodes the working directory from each directory name. Source: lib/sources/goose.py:25-70, lib/sources/grok.py:30-65
2. Normalise — Messages are coerced into a common schema (role, content, timestamp, model, cwd).
3. Upsert — Rows are written to the central messages table with INSERT OR REPLACE keyed on (source, session_id, ordinal). Frequency and message counts are recomputed; v2.3.0 fixed a sort-key crash where NULL frequency columns were compared against integers by switching the comparator to key.get("frequency") or 0. Source: lib/cmd_rebuild.py:140-180
4. Index — The FTS5 virtual table is refreshed, and if the sqlite-vec extension is available, embeddings produced by fastembed are written into a vec_messages virtual table.
5. Search / Refine — cmd_search.py issues BM25 or hybrid queries; cmd_refine.py optionally rewrites ambiguous rows through a configured LLM endpoint.

flowchart LR
    A[CLI clients on disk] --> B[Source adapters]
    B --> C[Normalised rows]
    C --> D[SQLite messages + FTS5]
    C --> E[fastembed vectors]
    E --> F[sqlite-vec table]
    D --> G[cmd_search / cmd_refine]
    F --> G
    G --> H[Results in terminal]

Dependencies and Extras

The package pins a minimal core and isolates optional integrations behind extras:

• Core (always installed): click/typer-style CLI plumbing, SQLite driver, fastembed>=0.4, sqlite-vec>=0.1.6. These were promoted from optional [vec] to core in v2.3.0 so vector and hybrid search work without an extra install step. Source: pyproject.toml:18-55
• Extras: [llm] pulls the OpenAI-compatible client used by lib/llm.py for refinement; [dev] adds pytest and lint tooling. The [vec] extra remains as a no-op alias for backwards compatibility. Source: pyproject.toml:40-70

Recent Changes Reflected in the Code

• v2.2.0 introduced the Goose and Grok CLI adapters, raising the supported source count to ten, and added matching docs/install/goose.md / docs/install/grok.md guides. Source: lib/sources/goose.py:1-30, lib/sources/grok.py:1-30
• v2.3.0 fixed the NoneType < int crash in rebuild sort keys and finalised the dependency reshuffle. Source: lib/cmd_rebuild.py:140-180

When to Use Total Recall

Use it as a long-term, local-first log of every AI assistant interaction on your machine. Use total-recall rebuild after installing a new CLI client or upgrading an existing one, total-recall search <query> for fast keyword lookup, and total-recall search --vec <query> for semantic retrieval once the embed index is populated. For bulk cleanup or noise reduction, total-recall refine runs an LLM pass against low-signal rows, optionally gated by a frequency or message-count threshold so that chatty sessions do not dominate the queue. Source: lib/cmd_search.py:20-90, lib/llm.py:15-60

Related Pages

Related topics: Overview, Lib

Lib

The lib/ package is the core extraction, storage, and search layer of total-recall. It isolates the concerns of pulling conversation history out of heterogeneous AI CLI clients, normalizing that history into a unified SQLite schema, and exposing full-text, vector, and hybrid retrieval over the resulting corpus. Application entry points under cmd/ and shell-level wrappers under hooks/lib/ delegate all I/O and parsing work to modules under lib/.

Purpose and Scope

lib/ exists so that every supported CLI client — 10 in total as of v2.2.0 — can be added by dropping a single adapter file into lib/sources/ without touching storage, search, or command code. Each adapter implements the same contract: given a local filesystem layout owned by a target CLI, enumerate session files, parse each session into a stream of role-tagged messages, and yield a normalized record set ready for SQLite insertion.

The package is also responsible for the search surface that total-recall advertises:

• Lexical search backed by SQLite FTS5.
• Vector search backed by sqlite-vec and fastembed, both promoted from optional [vec] extras to core dependencies in v2.3.0.
• Hybrid search combining the two paths with reciprocal-rank fusion.

Because the heavy deps now install by default, the hybrid/vector branches of lib/embeddings.py are reachable on a stock pip install with no extra selector. Source: lib/embeddings.py

Source Adapters (`lib/sources/`)

The lib/sources/ directory is the registry of client-specific parsers. New clients are added by introducing one module per client and re-exporting it through lib/sources/__init__.py, which is what command-level code imports.

Each adapter is expected to expose:

• A discover() function that locates the client's on-disk session store.
• A parse(path) generator yielding normalized message dicts (role, content, timestamp, cwd, session_id).
• A client_name constant used for tag/index naming.

Goose adapter (`lib/sources/goose.py`)

Goose stores sessions in a single SQLite database at ~/.local/share/goose/sessions/sessions.db rather than per-session files. The adapter opens the database read-only, walks the sessions and messages tables, and projects rows into the normalized message shape. Session working directories are recovered from a dedicated column. Source: lib/sources/goose.py

Because Goose stores everything in one file, the adapter short-circuits the usual per-file enumeration loop and emits a synthetic "file path" for downstream bookkeeping so the rest of the pipeline can treat Goose sessions identically to JSONL or JSON-backed clients.

Grok adapter (`lib/sources/grok.py`)

Grok's CLI uses JSONL chat history under ~/.grok/sessions/. The adapter streams each *.jsonl line, decodes the role/content pair, and recovers the original working directory by URL-decoding the enclosing directory name (Grok encodes cwd to keep it filesystem-safe). Source: lib/sources/grok.py

The URL-decoding step is the only Grok-specific normalization; once decoded, the adapter yields records indistinguishable from any other source, which is the design property the rest of lib/ depends on.

Storage, Search, and Rebuild Integration

lib/ owns the SQLite schema and the read paths that cmd/cmd_rebuild.py drives. During rebuild, each adapter feeds rows into a unified messages table; auxiliary columns such as frequency and message_count are populated by aggregation passes and are consumed by ranking and LLM refinement stages.

A crash reported against v2.2.x surfaced in the LLM refinement step: sort keys compared None against int because some aggregated rows contained NULL frequency or message_count values. The fix in v2.3.0 wraps those keys with or 0 so NULLs coerce to 0 instead of raising TypeError: '<' not supported between instances of 'NoneType' and 'int'. Source: cmd/cmd_rebuild.py

Search itself is a lib/ concern rather than a command concern:

Embeddings are produced lazily on rebuild; queries route through a single lib/embeddings.py entry point so callers do not branch on backend. Source: lib/embeddings.py

Extending `lib/`

Adding a new CLI client is a three-step operation confined to lib/sources/:

1. Create lib/sources/<client>.py implementing discover(), parse(), and client_name.
2. Register the adapter in lib/sources/__init__.py so command code can iterate it.
3. Add a documentation page under docs/install/<client>.md describing the on-disk layout the adapter consumes.

Because lib/ is the only layer that talks to client-specific data formats, the rest of the project — cmd/, hooks/lib/ — can stay client-agnostic. This boundary is what makes the v2.2.0 addition of Goose and Grok (and the future addition of further clients) a localized change rather than a cross-cutting refactor.

Related Pages

Related topics: Lib

Commands

Purpose and Scope

total-recall exposes its functionality through a set of named commands that act as the public operation surface for the assistant. Each command is documented as a standalone Markdown file under commands/ and is wired to a Python implementation inside lib/. Together they cover three concerns: (a) memory hygiene over previously ingested chat sessions, (b) operational telemetry such as spend and escalation handling, and (c) index lifecycle maintenance for the SQLite + sqlite-vec back end. Source: commands/recall-check-banned.md:1-1, commands/recall-rebuild.md:1-1.

The command surface is intentionally narrow. A user invokes a command by name; the dispatcher in lib/commands/__init__.py resolves the slug, loads the corresponding handler, and forwards CLI arguments. This keeps each command independently testable and lets new commands be added without touching unrelated modules. Source: lib/commands/__init__.py:1-1.

Memory Workflow Commands

The memory-oriented commands mutate or audit the persisted corpus of messages harvested from the supported CLI clients.

• recall-check-banned scans stored messages against a blocklist and reports hits without deleting them, giving the operator a chance to inspect before any purge runs. Source: commands/recall-check-banned.md:1-1.
• recall-corrections records user-supplied corrections and links them to the originating message IDs so that future retrieval can prefer corrected phrasing. Source: commands/recall-corrections.md:1-1.
• recall-decisions captures decision rationales tagged during a session, producing a queryable log of "why we chose X" notes distinct from raw transcripts. Source: commands/recall-decisions.md:1-1.
• recall-goal reads or updates the active goal that scopes retrieval, so subsequent searches bias toward a stated objective. Source: commands/recall-goal.md:1-1.

These four commands share a common pattern: they take an identifier (message ID, session ID, or goal slug) plus optional flags, then perform an idempotent write or read against the database. They never re-ingest sessions from the upstream CLI clients; that responsibility belongs to the source adapters in lib/sources/. Source: lib/sources/goose.py:1-1, lib/sources/grok.py:1-1.

Operational Commands

The operational commands surface accounting and maintenance behaviour.

recall-cost reads the per-message counters written during ingest and rolls them up by source (e.g., Goose, Grok, plus the remaining eight adapters bringing the total to ten CLI clients). Source: commands/recall-cost.md:1-1, lib/sources/goose.py:1-1.

recall-rebuild is the heaviest command and the one most likely to exercise edge cases. It iterates every session, re-computes frequency and message_count columns, and reissues vec embeddings via fastembed. As of v2.3.0, the sort keys used to order rows during refinement were hardened: cmd_rebuild.py now evaluates or 0 on those keys so that SQLite NULL values no longer raise '<\' not supported between instances of \'NoneType\' and \'int\' during the LLM refinement stage. Source: commands/recall-rebuild.md:1-1, lib/cmd_rebuild.py:1-1.

recall-escalation is the bridge between automated capture and human oversight; it writes an escalation row and emits a notification hook so downstream tooling can pick the session up. Source: commands/recall-escalation.md:1-1.

Dependency and Packaging Notes

Command behaviour is conditional on which optional extras are installed. Until v2.2.0, vec/hybrid search and LLM refinement lived behind [vec] and [llm] extras. In v2.3.0 those dependencies — fastembed>=0.4 and sqlite-vec>=0.1.6 — were promoted to core requirements, so recall-rebuild and vector-aware queries work out of the box. Operators who do not want a network-touching dependency can still pin older versions, but the documented happy path no longer requires an pip install total-recall[vec,llm] step. Source: commands/recall-rebuild.md:1-1.

Invocation Pattern

Every command follows the same shape once dispatched by lib/commands/__init__.py:

recall <command> [--source <adapter>] [--since <date>] [--json]

Flags are not shared globally; each handler parses only the subset it advertises in its companion Markdown file. New commands should add their own flag set, document them in the matching commands/recall-*.md, and register the handler in lib/commands/__init__.py. This convention keeps the command catalogue self-describing and matches the structure used by all eleven files listed above. Source: lib/commands/__init__.py:1-1, commands/recall-check-banned.md:1-1, commands/recall-cost.md:1-1.

Related Pages

• See the Sources page for the per-adapter details behind commands like recall-cost and recall-rebuild.
• See the Configuration page for environment variables that toggle command behaviour.
• See the Release Notes page for the v2.2.0 and v2.3.0 changes referenced above.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/88plug/total-recall

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/88plug/total-recall

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/88plug/total-recall

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/88plug/total-recall

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/88plug/total-recall

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/88plug/total-recall

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/88plug/total-recall

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using total-recall with real data or production workflows.

• v2.3.18 - github / github_release
• v2.3.0 - github / github_release
• v2.2.0 - github / github_release
• Configuration risk requires verification - GitHub / issue