Related Pages

Related topics: MCP Server Architecture and Toolset, Storage, Persistence, Privacy, and Evaluation

Introduction and Quick Start

ShapeLex is a local memory layer designed to compress and persist conversational or textual context, allowing agents and applications to reclaim token budget without losing semantic information. This page introduces the project scope, the v0.5.0 release highlights, and a minimal workflow that gets a developer from clone to a first successful round-trip of compression and expansion.

What ShapeLex Is

ShapeLex positions itself as a safer, lower-overhead local memory layer rather than a full retrieval-augmented generation system. It is built around two primitives:

• Compression — turns a raw string into a compact, storable handle.
• Expansion — resolves a handle back into the original (or near-original) string.

The project began as an initial prototype and matured into v0.5.0 with safety and observability improvements. The README frames the project around agent memory and token economy, while the QUICKSTART documents describe the end-to-end onboarding flow for new users in both English and Spanish. Source: README.md:1-40.

v0.5.0 Highlights

The v0.5.0 release focuses on safety, file-backed persistence, and observability. Three notable additions shape the public surface:

The heuristic-estimator label is important: it tells consumers that compression ratios are estimates produced by a heuristic, not exact measurements, which prevents false precision in downstream metrics. Source: docs/AGENT_INSTRUCTIONS.md:1-60.

Minimal Workflow

A typical first use of ShapeLex follows four steps:

1. Install or vendor the package locally. The QUICKSTART guides users through this without requiring a remote service.
2. Call shapelex_compress_text with the text you want to store. Optionally pass sourcePath to bind the handle to a workspace file rather than relying solely on in-memory state.
3. Persist the returned handle in your application state, prompt, or session memory.
4. Call shapelex_expand_text later to reconstruct the original text. If the bound source file has changed since compression, the checksum check invalidates the handle rather than returning stale data.

The QUICKSTART documents are intentionally bilingual; docs/QUICKSTART.md is the English version and docs/QUICKSTART.es.md mirrors the same flow for Spanish-speaking users, ensuring the onboarding experience is symmetric. Source: docs/QUICKSTART.md:1-80.

Operational Notes for New Users

Two operational details matter when starting out:

• Heuristic estimates are labeled, not exact. Any ratio or savings number you see from telemetry is produced by a heuristic; treat it as a planning aid, not a billing-grade measurement.
• Workspace-bound compression is the safer default. When the text originates from a file in your project, prefer shapelex_compress_text.sourcePath so that downstream expansion can detect modifications via checksum and refuse to silently return drifted content.

The agent-oriented guides (docs/AGENT_SETUP_PROMPT.md and docs/AGENT_INSTRUCTIONS.md) describe how assistants should phrase these operations and which safeguards to surface to end users. Source: docs/AGENT_SETUP_PROMPT.md:1-60.

flowchart LR
  A[Raw text] --> B[shapelex_compress_text]
  B -- sourcePath --> C[Bound file + checksum]
  B --> D[Handle]
  D --> E[shapelex_expand_text]
  C -- verify --> E
  E --> F[Original or invalidated]

Where to Go Next

After completing the quick start, the recommended next readings are:

• The v0.5.0 release notes in the README for any breaking changes removed from earlier prototypes.
• The full QUICKSTART for parameter-level guidance on shapelex_compress_text and shapelex_expand_text.
• The agent instructions document for safe phrasing patterns when ShapeLex is exposed through an LLM-driven interface.

Together, these documents establish ShapeLex v0.5.0 as a bounded, inspectable local memory layer suitable as a foundation for agent workflows. Source: README.md:1-120.

Related Pages

Related topics: Introduction and Quick Start, Fingerprint Retrieval, Hierarchy, and Token Accounting, Storage, Persistence, Privacy, and Evaluation

MCP Server Architecture and Toolset

The ShapeLex MCP (Model Context Protocol) server exposes the project's local memory layer as a set of tools that AI agents can invoke over stdio. It acts as the thin transport-and-routing shell that wraps the core compression/expansion engine defined in src/shapelex.ts, turning it into a discoverable, schema-validated service for MCP-compatible clients such as Claude Desktop.

1. Entry Point and Process Lifecycle

The server is launched as a Node.js child process through bin/shapelex-mcp.js, which is registered as the bin executable in package.json under the shapelex-mcp command. This entry point boots the runtime, instantiates the server class from src/mcp-server.ts, connects it to a StdioServerTransport, and keeps the process alive for the duration of the agent session.

Source: bin/shapelex-mcp.js:1-30

Source: package.json:18-26

Because all I/O happens over standard input/output, the server inherits the same single-writer assumption used by the rest of the workspace: the working directory passed by the agent (or defaulted to process.cwd()) becomes the implicit workspace root for any file-backed compression.

2. Server Initialization and Capability Negotiation

Inside src/mcp-server.ts, the server class extends the MCP SDK Server, names itself "shapelex", and advertises its version by reading src/version.ts. During construction it declares the tools capability and registers its tool catalog with setRequestHandler(ListToolsRequestSchema, ...). Each tool entry contains a human-readable description and a Zod-derived JSON Schema that the client uses to validate arguments before invocation.

Source: src/mcp-server.ts:12-58

Source: src/version.ts:1-6

Initialization is split into two phases: init() (called by the entry script) wires the CallToolRequestHandler and ListToolsRequestHandler, and run() opens the stdio transport and awaits shutdown. Errors during tool execution are caught per-call and returned as isError: true content blocks rather than thrown, so a single malformed request cannot crash the agent session.

3. Toolset Catalog

The toolset is deliberately small and maps one-to-one onto the public operations of the core engine. Each handler delegates to a method on a shared ShapeLex instance rather than re-implementing logic, keeping the MCP layer a pure adapter.

shapelex_compress_text is the only tool that accepts sourcePath. When provided, the handle is bound to the file's contents at compression time and a checksum is recorded so that subsequent shapelex_expand calls can detect edits and refuse to return stale data. This is the mechanism referenced in the v0.5.0 release notes as "checksum-verified expansion so changed source files invalidate stale handles."

Source: src/mcp-server.ts:60-140

Source: src/shapelex.ts:40-95

4. Compression Telemetry and Heuristic Labeling

shapelex_stats returns a JSON object summarizing the cumulative counts for the current process: total compression requests, total bytes in, total bytes out, mean compression ratio, and the number of invalidated handles. The payload also carries an explicit estimator field whose value is the string "heuristic" to make clear to downstream consumers that ratios are computed from observed lengths, not from any formal information-theoretic bound.

This label was introduced in v0.5.0 to satisfy the release-note commitment of "cumulative per-session compression telemetry with an explicit heuristic-estimator label," and prevents agents from misinterpreting the ratio as a guaranteed upper bound.

Source: src/mcp-server.ts:141-180

Source: src/shapelex.ts:120-160

5. Skill Integration and Discoverability

The MCP server is paired with a skill manifest at skills/shapelex-memory/SKILL.md that describes the toolset in agent-natural language. When a host loads the skill, it learns when to prefer compression (long, repetitive, file-bound passages) versus when to pass content through directly, which reduces unnecessary round-trips and keeps the toolset aligned with the documented usage patterns discussed in the community.

Source: skills/shapelex-memory/SKILL.md:1-45

Data Flow Summary

Agent → stdio → ListToolsRequestSchema → catalog
Agent → stdio → CallToolRequestSchema  → handler → ShapeLex core → response
ShapeLex core → per-session counters → shapelex_stats payload (estimator: "heuristic")

The architecture is intentionally linear: every request enters through the MCP handler, funnels into the core, and exits as a structured content block. There is no background worker, no network listener, and no shared mutable state across processes, which keeps the server safe to embed alongside other MCP servers and bounded to a single workspace per invocation.

Source: src/mcp-server.ts:1-200

Source: src/shapelex.ts:1-200

Related Pages

Related topics: MCP Server Architecture and Toolset, Storage, Persistence, Privacy, and Evaluation

Fingerprint Retrieval, Hierarchy, and Token Accounting

The fingerprint subsystem is ShapeLex's local memory layer that converts raw text into a normalized, token-aligned representation, groups those tokens into reusable profiles, and exposes a retrieval API that can reconstruct or extend compressed text within a single workspace. This page documents how fingerprints are retrieved, how their hierarchy is structured, and how token accounting is performed during compression and expansion. It reflects the v0.5.0 release notes that added cumulative per-session compression telemetry and explicit heuristic-estimator labeling.

Purpose and Scope

The fingerprint module exists to provide a deterministic, low-overhead way to recognize previously seen text fragments and to translate between compressed handles and their original expansions. It is consumed by shapelex_compress_text and its expansion counterpart through workspace-bound handles (shapelex_compress_text.sourcePath) and checksum-verified round-trips that invalidate stale handles when the source file changes. Within that boundary, the module is responsible for three concerns: tokenization, profile construction and lookup, and per-session telemetry. It deliberately stops short of performing LLM-style semantic retrieval and instead relies on lexical and normalized matching.

Source: src/fingerprint/index.ts:1-40

Hierarchy and Data Types

Fingerprints are organized in a fixed three-level hierarchy: raw text is broken into tokens, tokens are grouped into profiles, and profiles are referenced by stable handles. The shape of each level is defined in types.ts, and the runtime constructors live in profile.ts.

The hierarchy is intentionally flat at the public API surface: callers only ever exchange FingerprintHandle values, while the internal profile bag is rebuilt on demand by the matcher. The FingerprintToken carries both an identifier and a normalized form so that downstream code can audit what was actually compressed without re-running tokenization.

Source: src/fingerprint/types.ts:1-80

Profiles are constructed by buildProfileFromTokens in profile.ts, which validates that the token list is non-empty, computes a content checksum, and stamps the session id onto the resulting FingerprintProfile. The checksum is later used during expansion to confirm that the source file referenced by shapelex_compress_text.sourcePath has not changed.

Source: src/fingerprint/profile.ts:1-60

Retrieval and Matching Flow

Retrieval follows a normalize → match → resolve pipeline. The normalize.ts module owns text canonicalization (case folding, whitespace collapsing, and Unicode normalization), which guarantees that two strings that differ only in trivial formatting collapse to the same token stream. tokenize.ts then segments the normalized string and assigns stable ids, calling into profile.ts to either reuse an existing FingerprintProfile or create a new one.

The matcher is the orchestrator that callers interact with most often. Given a query handle, it loads the referenced profile, runs the normalized tokens against the in-memory index, and returns the best-scoring candidates along with their reconstructed surface forms. A high-level flow looks like this:

flowchart LR
  A[Input text or handle] --> B[normalize.ts]
  B --> C[tokenize.ts]
  C --> D{Profile exists?}
  D -- yes --> E[matcher.ts]
  D -- no --> F[profile.ts build]
  F --> E
  E --> G[Reconstructed text + telemetry]

Source: src/fingerprint/matcher.ts:1-120

The matcher also enforces workspace boundaries: a profile built from one sourcePath cannot be retrieved through a handle scoped to another workspace, which is what makes the v0.5.0 "workspace-bound file-backed compression" guarantee meaningful in practice.

Source: src/fingerprint/index.ts:40-90

Token Accounting

Token accounting is the subsystem's main cost-control surface and is what the v0.5.0 release notes describe as "cumulative per-session compression telemetry with an explicit heuristic-estimator label". Every call into compressText increments counters on FingerprintSessionState: total input tokens, compressed tokens (tokens that hit an existing profile), and the estimator label (e.g. heuristic-v1) that was used to project the compression ratio.

interface FingerprintSessionState {
  sessionId: string;
  estimatorLabel: string;
  totalTokens: number;
  compressedTokens: number;
  profilesCreated: number;
  profileReuses: number;
}

The numbers are accumulated, not per-call, which lets callers reason about savings across a whole session rather than per snippet. During expansion, the same state object is updated with expansionTokens so the ratio can be reported as (compressedTokens + expansionTokens) / totalTokens. If a checksum mismatch is detected because the source file referenced by sourcePath changed, the expansion is aborted and the stale handle is invalidated, which preserves the integrity of the running totals.

Source: src/fingerprint/tokenize.ts:1-70 Source: src/fingerprint/profile.ts:60-110 Source: src/fingerprint/types.ts:60-100

This accounting model is what allows ShapeLex to advertise "lower-overhead local memory layer" in v0.5.0: callers can inspect FingerprintSessionState directly, decide when to flush profiles, and observe the estimator label to know which heuristic produced the reported ratios.

Related Pages

Related topics: MCP Server Architecture and Toolset, Fingerprint Retrieval, Hierarchy, and Token Accounting

Storage, Persistence, Privacy, and Evaluation

Purpose and Scope

ShapeLex v0.5.0 positions the project as a safer, lower-overhead local memory layer rather than a network service. This page documents the four intersecting concerns that make that positioning concrete: how compressed artifacts are stored and reloaded (src/storage/index.ts, src/storage/store-v2.ts), how protocol-shaped operations are recorded (src/protocol-ledger.ts), what stays on-device versus what gets emitted (src/shapelex-doctor.ts), and how the project validates that the layer still behaves correctly across releases (src/shapelex-smoke-eval.ts, src/shapelex-e2e-eval.ts).

Together these modules define ShapeLex's contract with the host environment: data lives inside an explicit workspace boundary, integrity is checked on load, telemetry is session-scoped, and correctness is enforced by a two-tier evaluation suite.

Storage and Persistence

The persistence layer is exposed through src/storage/index.ts, which is the public entry point that callers see. The v2 backend in src/storage/store-v2.ts implements the actual read/write path. In v0.5.0, persistence was tightened around three guarantees:

• Workspace-bound handles. Compressed artifacts created via shapelex_compress_text.sourcePath are anchored to a specific workspace file path. Storing the source path alongside the compressed blob prevents accidental cross-file reuse Source: src/storage/index.ts:1-120.
• File-backed compression. Compression can target a file rather than only an in-memory string, allowing large inputs to be streamed through the compressor without buffering the full payload in RAM Source: src/storage/store-v2.ts:1-160.
• Checksum-verified expansion. On expand, the implementation recomputes a checksum over the source path and compares it against the value captured at compression time. A mismatch invalidates the handle and forces a recompress, so stale caches cannot silently resurrect outdated content Source: src/storage/store-v2.ts:160-280.

Handles returned by these functions are intentionally opaque: they identify a (workspace, path, checksum, compression-tuple) quadruple rather than exposing internal offsets, which simplifies invalidation and makes the storage layer safe to expose to higher-level tooling Source: src/storage/index.ts:120-220.

Protocol Ledger and Session Boundaries

The protocol ledger in src/protocol-ledger.ts is the recording surface that sits on top of storage. Each call into ShapeLex — compression, expansion, invalidation — is appended as a ledger entry together with the originating session id. This is what gives ShapeLex its per-session character: a session is the unit of visibility for telemetry, and the ledger is what makes that unit machine-checkable.

The ledger is append-only and stored locally; it is the source of truth for the per-session compression telemetry surfaced in v0.5.0. Because it lives next to the storage layer, deleting or rotating a workspace implicitly retires its ledger slice, which preserves the "local memory layer" invariant Source: src/protocol-ledger.ts:1-140.

Privacy Boundaries

ShapeLex treats privacy as a workspace-boundary property rather than a per-field one. There is no remote service to redact against; instead, the system guarantees that:

• All artifacts and ledger entries are written under a caller-supplied workspace root, so a host application can sandbox the layer using its own filesystem permissions Source: src/storage/index.ts:40-90.
• Source paths and checksums are the only metadata persisted; raw input text is not stored in a separately indexable form once compression completes Source: src/storage/store-v2.ts:200-260.
• Telemetry emitted by src/shapelex-doctor.ts is cumulative and per-session with an explicit heuristic-estimator label, signalling that the numbers are estimates derived from local counters rather than ground-truth measurements from a central service Source: src/shapelex-doctor.ts:1-180.

The heuristic-estimator label is important: it tells downstream consumers — and reviewers reading evaluation output — that compression ratios and token savings reported by the doctor are approximations, suitable for trend tracking but not for billing or SLA assertions.

Evaluation Suite

Correctness of the storage and privacy invariants is enforced by two complementary runners:

• src/shapelex-smoke-eval.ts performs fast, deterministic checks on every invocation. It verifies that compress/expand round-trip preserves content, that checksum mismatch invalidates a handle, and that sourcePath handles reject cross-workspace access attempts Source: src/shapelex-smoke-eval.ts:1-220.
• src/shapelex-e2e-eval.ts runs a longer, scenario-driven suite that exercises realistic sequences — repeated edits to a source file, sequential sessions, ledger rotation, and doctor output stability — to confirm that the invariants hold end-to-end rather than only in isolation Source: src/shapelex-e2e-eval.ts:1-260.

The two-tier design lets contributors gate every change on the smoke suite while reserving the slower e2e run for release validation, which is what makes the v0.5.0 storage hardening practical to merge without a dedicated QA environment Source: src/shapelex-smoke-eval.ts:200-260, Source: src/shapelex-e2e-eval.ts:240-300.

Cross-Cutting Workflow

The typical lifecycle of a compressed artifact is: caller invokes shapelex_compress_text with a sourcePath → store-v2.ts hashes the file and writes the compressed blob alongside a handle → the protocol ledger records the operation against the active session → expand re-hashes the source path and either returns the cached blob or recompresses on mismatch → shapelex-doctor aggregates per-session counters from the ledger into a heuristic-estimator telemetry snapshot → evaluation runners replay the flow to confirm invariants. The privacy story follows from the same chain: because every step reads only the source path and writes only into the workspace, no step requires network access or external trust Source: src/storage/index.ts:200-260, Source: src/protocol-ledger.ts:140-220, Source: src/shapelex-doctor.ts:180-260.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/AldoDior/shapelex

2. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/AldoDior/shapelex

3. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/AldoDior/shapelex

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/AldoDior/shapelex

5. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/AldoDior/shapelex

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/AldoDior/shapelex

7. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/AldoDior/shapelex

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using shapelex with real data or production workflows.

• ShapeLex v0.5.0 - github / github_release
• ShapeLex v0.4.0 - github / github_release
• Configuration risk requires verification - GitHub / issue