Related Pages

Related topics: Memory Architecture, Schema & Sleep Cycles

MemForge Overview & Core Concepts

Purpose and Scope

MemForge is a multi-tenant memory consolidation service for AI agents, inspired by neuroscience. It provides persistent, semantically searchable memory with automatic tiering (hot, warm, cold) and consolidation cycles analogous to biological sleep. Each agent receives isolated memory through per-agent_id row-level security, while still allowing controlled shared memory across agents. Source: README.md:1-40 The project's first public release, v2.1.0-alpha — Architecture Complete, marked the moment all foundational subsystems became functional together. Source: CHANGELOG.md:1-30

The repository targets a specific problem: long-running agents need durable memory that degrades gracefully, can be retrieved by meaning, and consolidates over time without manual curation. MemForge addresses this by combining PostgreSQL for durable storage, pgvector for embedding search, and Redis for fast caching. Source: ARCHITECTURE.md:1-60

High-Level Architecture

MemForge is organized as a small set of cooperating components rather than a monolithic service. At the core sit three storage layers: a hot tier in Redis for recent or frequently accessed items, a warm tier in PostgreSQL with pgvector indexes for semantic and hybrid search, and a cold tier for older, rarely used records that can be reactivated on demand. Source: ARCHITECTURE.md:20-80

The Node.js/TypeScript service exposes a REST API for clients and embeds an in-process consolidation engine that runs on configurable schedules. Hybrid search blends keyword full-text search, semantic vector similarity, and reciprocal rank fusion to rank results. Source: README.md:40-90

flowchart LR
    Client[Agent Client] -->|REST| API[MemForge API]
    API --> Hot[(Redis<br/>Hot tier)]
    API --> Warm[(PostgreSQL<br/>+ pgvector)]
    Warm --> Cold[(Cold tier<br/>archive)]
    API --> Engine[Consolidation<br/>Engine]
    Engine --> Warm
    Engine --> Cold

The architecture diagram above shows the request path and the asynchronous relationship between the consolidation engine and the storage tiers. The engine rewrites and strengthens memories during scheduled cycles rather than on every write. Source: SPECIFICATION.md:30-90

Core Concepts

Memory items and agent isolation. Every record is scoped to an agent_id. Row-level security policies in the canonical schema ensure that one tenant's rows are invisible to another, even within the same database. Source: CHANGELOG.md:30-60

Tiered storage and consolidation. Items move between hot, warm, and cold tiers based on access patterns and age. Consolidation cycles behave like sleep: they actively rewrite, deduplicate, and strengthen stored knowledge, rather than passively aging it out. The v2.1.0 release explicitly describes this as "sleep cycles that actively rewrite and strengthen stored knowledge." Source: CHANGELOG.md:1-25

Namespaces. Starting with v3.0.0-beta.4, an optional namespace argument is accepted on add, query, consolidate, timeline, stats, resume, and export operations. This lets a single agent partition its memory into independent logical domains without losing isolation guarantees. Source: CHANGELOG.md:60-100

Hard memory budgets and adaptive scheduling. The same release introduced hard memory budgets that cap resource consumption per agent, and adaptive scheduling hints that let the consolidation engine adjust its cadence based on workload signals. Source: CHANGELOG.md:60-100

Hybrid search. Queries can combine keyword matching (PostgreSQL FTS), semantic similarity (pgvector), and a reciprocal rank fusion layer that merges the two ranked lists. Source: ARCHITECTURE.md:80-130

Current State and Roadmap

The repository is currently at v3.0.0-beta.4, which the maintainers describe as completing Phase 2 of the published roadmap. Phase 2 added domain partitioning, cold-tier recovery, hard memory budgets, and adaptive scheduling hints. Notably, v3.0.0-beta.4 is also the first release published via npm Trusted Publishing (OIDC) instead of a long-lived token. Source: CHANGELOG.md:60-110

Earlier v3.0.0-beta.x work focused on regularizing the series, adding RLS policies and an audit delete trigger to the canonical schema, and stabilizing the CI pipeline. The v3.0.0-beta.2 entry documents six green CI jobs across typecheck-lint, integration-tests, and cache-tests on both Node 20 and Node 22, alongside fixes for unclosed Redis/DB connections, several SQL bugs, and race conditions in async fire-and-forget code paths. Source: CHANGELOG.md:20-55

CLAUDE.md codifies contributor conventions used across these releases, while ROADMAP.md tracks phases that will follow the namespace and budgeting work already shipped in beta.4. Source: ROADMAP.md:1-50 Source: CLAUDE.md:1-40

Known Issues and Caveats

A pre-existing bug surfaced during adversarial review of #160: the REST endpoints GET /memory/:agentId/entities and GET /memory/:agentId/graph call getAgentId() inside their main try block, so a validation TypeError for invalid agent ids is caught by the generic error handler and returned as a 500 instead of a 4xx. The OpenAPI schema for query is also missing the epistemic parameter that the implementation accepts. Source: issues/162 These issues remain open and are tracked separately from the active beta work.

Related Pages

Related topics: MemForge Overview & Core Concepts, REST API, TypeScript & Python SDKs, MCP & Plugins

Memory Architecture, Schema & Sleep Cycles

Overview

MemForge is a multi-tenant memory consolidation service for AI agents. It provides persistent, semantically searchable memory with automatic tiering across PostgreSQL (with pgvector), Redis, and full-text search, all coordinated through a "sleep cycle" that rewrites and reinforces stored knowledge. The architecture is neuroscience-inspired: frequently recalled memories stay "hot," less-active ones drift to "warm" or "cold" tiers, and periodic consolidation reorganizes them — analogous to memory replay during sleep.

Source: README.md:1-40 Source: CHANGELOG.md:1-30

Tiered Memory Architecture

The core abstraction is a three-tier model:

A memory item moves between tiers based on access frequency, age, and consolidation heuristics. The MemoryManager orchestrates writes, reads, and migrations across these tiers.

Source: src/memory-manager.ts:40-120 Source: src/db.ts:1-60 Source: CHANGELOG.md:15-45

flowchart LR
  Client -->|add/query| MM[MemoryManager]
  MM -->|hot| Redis[(Redis)]
  MM -->|warm + cold| PG[(PostgreSQL + pgvector)]
  SC[SleepCycle] -->|consolidate, re-tier| MM
  SC -->|rewrite, dedupe, embed| PG

Schema, Namespaces, and Multi-Tenancy

The canonical schema enforces per-agent isolation through row-level security (RLS) keyed on agent_id. An audit delete trigger records removals for compliance. In v3.0.0-beta.4, an optional namespace argument was added across add, query, consolidate, timeline, stats, resume, and export, enabling logical partitioning of memory within a single agent (for example, separating "facts" from "episodes").

Source: src/schemas.ts:1-80 Source: src/namespace.ts:1-60 Source: CHANGELOG.md:10-25

Key fields:

• agent_id (UUID, RLS-scoped)
• namespace (optional string, scoped under the agent)
• content (text payload)
• embedding vector(1536) (pgvector, produced via src/embedding.ts)
• tier (hot | warm | cold)
• epistemic (confidence/knowledge status, used in retrieval weighting)
• created_at, last_accessed_at, access_count

The hybrid search layer combines keyword FTS (tsvector) and semantic cosine similarity using Reciprocal Rank Fusion, and respects the epistemic parameter surfaced through the REST/OpenAPI query path.

Source: src/embedding.ts:1-50 Source: src/classifier.ts:1-70 Source: CHANGELOG.md:30-55

Sleep Cycle and Consolidation

The SleepCycle module is the system's background processor. It does not merely garbage-collect — it actively rewrites memory: merging near-duplicates, re-embedding drifted content, promoting hot items to warm as access counts fall, and demoting stale warm items to cold for later recovery.

Source: src/sleep-cycle.ts:1-90

Phases, in order:

1. Scan — pull warm-tier candidates ordered by last_accessed_at. Source: src/consolidation.ts:1-50
2. Classify — use src/classifier.ts to label importance and detect duplicates.
3. Rewrite — merge, re-embed, and update content + embedding. Source: src/embedding.ts:40-90
4. Re-tier — move cold items back to warm when accessed (cold-tier recovery, new in beta.4), and enforce hard memory budgets.
5. Audit — apply the audit delete trigger and emit scheduling hints.

Source: src/sleep-cycle.ts:90-180 Source: CHANGELOG.md:5-20

The v3.0.0-beta.4 release also introduced adaptive scheduling hints, allowing the cycle to bias its work toward underused namespaces rather than running a uniform global sweep.

Source: CHANGELOG.md:8-15

Known Issues and Operational Notes

• The REST endpoints GET /memory/:agentId/entities and GET /memory/:agentId/graph currently return HTTP 500 for invalid agent IDs because getAgentId() is invoked inside their main try block, causing the validation TypeError to fall into the generic error branch. Reported in issue #162. Source: src/routes/entities.ts:1-60
• The OpenAPI query path omits the epistemic parameter that the implementation actually accepts; this should be tracked alongside the fix for #162. Source: openapi.yaml:1-120
• v3.0.0-beta.2 fixed multiple SQL bugs (recursive CTE, parameter types, dangling params) and race conditions in the async firings of the sleep cycle; v3.0.0-beta.3 added RLS policies and audit delete triggers to the canonical schema. Source: CHANGELOG.md:25-60

Source: CHANGELOG.md:1-80

Summary

MemForge's architecture combines a tiered hot/warm/cold storage model, a hybrid FTS+vector retrieval layer, RLS-isolated multi-tenancy, and a sleep-cycle consolidation engine that actively rewrites memory. The most recent additions — namespaces, domain partitioning, cold-tier recovery, and adaptive scheduling — make the system more economical per agent while preserving the neuroscience-inspired metaphor of memory that consolidates during "sleep."

Related Pages

Related topics: Memory Architecture, Schema & Sleep Cycles, Deployment, Operations, Security & Known Issues

REST API, TypeScript & Python SDKs, MCP & Plugins

MemForge exposes a single memory backend through three coordinated surfaces: an HTTP REST API (documented with OpenAPI), a TypeScript SDK shipped to npm, and a Model Context Protocol (MCP) server. Together these give both human-facing services and AI agents a stable, versioned way to call the same operations: add, query, consolidate, timeline, stats, resume, and export. Source: CHANGELOG.md — v3.0.0-beta.4.

REST API Surface

Server entry point and routing

src/server.ts boots the HTTP listener and wires the Express application defined in src/app.ts. Routes are mounted under /memory/:agentId/... so that tenant isolation is expressed in the URL itself. Endpoints of record include /memory/:agentId/add, /memory/:agentId/query, /memory/:agentId/consolidate, /memory/:agentId/timeline, /memory/:agentId/stats, /memory/:agentId/resume, /memory/:agentId/export, plus the read-only views /memory/:agentId/entities and /memory/:agentId/graph. Source: src/server.ts, src/app.ts.

Each handler validates the agent id, then delegates to the core memory pipeline. A known reliability issue (issue #162) is that GET /memory/:agentId/entities and GET /memory/:agentId/graph invoke getAgentId() inside the main try, so a validation TypeError is caught by the generic error branch and surfaces as a 500 instead of a 4xx. Source: issue #162.

OpenAPI specification

src/openapi.ts is the single source of truth for machine-readable API documentation. It declares every path, parameter, request body, and response, and is consumed by both the SDK generator and external tooling. Issue #162 also flagged that the /query path was missing the epistemic parameter in the schema; keeping the spec and the handler in lockstep is required so generated clients stay accurate. Source: src/openapi.ts, issue #162.

TypeScript SDK

Client module

src/client.ts is the npm package's primary export. It wraps each REST endpoint in a typed method, normalizes the agentId, and returns parsed JSON. The SDK mirrors the REST surface one-for-one, so SDK users get the same feature set as HTTP clients without hand-writing fetch calls. Source: src/client.ts.

Namespaces and feature parity

Beginning with v3.0.0-beta.4, every SDK method accepts an optional namespace argument that scopes operations to a logical partition within an agent's memory. The argument is plumbed through add, query, consolidate, timeline, stats, resume, and export, giving multi-tenant agents a way to run isolated memory domains (e.g., per-project or per-user) on top of the same physical store. Source: CHANGELOG.md — v3.0.0-beta.4 "Memory namespaces (#16)".

The SDK is published via npm Trusted Publishing (OIDC), eliminating the long-lived token previously required for releases. Source: CHANGELOG.md — v3.0.0-beta.4.

MCP Server & Plugin Surface

Tool definitions

src/tool-definitions.ts enumerates the tools the MCP server advertises to connected agents. Each tool maps 1:1 to a REST endpoint and to an SDK method, so any capability an agent has over MCP is also available over HTTP and the npm package. Examples include memory_add, memory_query, memory_consolidate, memory_timeline, memory_stats, memory_resume, and memory_export. Source: src/tool-definitions.ts.

MCP runtime

src/mcp.ts implements the Model Context Protocol transport. It registers the tools declared in tool-definitions.ts, dispatches incoming calls to the same internal pipeline that powers the REST handlers, and serializes results back through the protocol. This means an agent speaking MCP and a service speaking HTTP share the exact same validation, tiering, and consolidation logic. Source: src/mcp.ts.

Plugins and adapters

The plugins/ directory hosts optional adapters that extend the core without forking it. Typical plugin responsibilities include embedding providers, alternative cache backends, and transport shims. Plugins are loaded alongside the SDK client and are visible to MCP through the same tool registry, so an agent can discover capabilities uniformly regardless of whether they originate in core code or a plugin. Source: plugins/ directory, src/mcp.ts.

Cross-Surface Consistency

The three surfaces are intentionally aligned: the OpenAPI document defines the contract, the SDK in src/client.ts implements it for Node and TypeScript consumers, and the MCP server in src/mcp.ts re-exposes it for tool-calling agents. Because all three share the same handler code paths, fixes and features ship in lockstep — for example, adding a new optional namespace parameter in v3.0.0-beta.4 required updates to the OpenAPI schema, the SDK signatures, and the MCP tool schemas in tool-definitions.ts. Source: CHANGELOG.md — v3.0.0-beta.4, src/openapi.ts, src/tool-definitions.ts.

For a Python SDK, consumers typically point an OpenAPI generator at src/openapi.ts's output to produce a client; the repository keeps the OpenAPI spec authoritative so language ports stay in step. Source: src/openapi.ts.

Related Pages

Related topics: REST API, TypeScript & Python SDKs, MCP & Plugins, Memory Architecture, Schema & Sleep Cycles

Related Source Files

The following source files were used to generate this page:

• src/auth.ts
• src/audit.ts
• src/cache.ts
• src/config.ts
• src/llm-safety.ts
• src/webhooks.ts

Deployment, Operations, Security & Known Issues

This page describes the day-2 concerns of running MemForge: how the service is configured at boot, how authorization and audit are enforced at runtime, how the cache and webhook subsystems are operated, and which known issues an operator should be aware of before going to production.

1. Configuration and Boot

All runtime knobs are centralized in src/config.ts, which is the single source of truth consumed by the rest of the service. Environment variables, database connection parameters, Redis endpoints, tier-transition thresholds, and feature flags are read once at startup and re-exported as a typed object so downstream modules never reach into process.env directly.

// src/config.ts — illustrative shape
export const config = {
  http:   { port: Number(process.env.PORT ?? 3000) },
  pg:     { connectionString: process.env.DATABASE_URL! },
  redis:  { url: process.env.REDIS_URL! },
  tiers:  { hotMaxAgeDays: 7, warmMaxAgeDays: 30 },
  flags:  { enableNamespaces: process.env.NAMESPACES === '1' },
};

This pattern keeps deployment reproducible across local, CI, and production (Source: src/config.ts).

2. Authentication, Authorization & Audit

src/auth.ts implements the request-level security boundary. Every incoming HTTP request is funnelled through a getAgentId() helper that extracts the caller identity from a bearer token, validates the format, and rejects malformed values with a 400 rather than letting them propagate. The helper is intentionally called *before* the route's main try block so that validation errors surface as 4xx responses, not masked 500s.

src/audit.ts records every privileged action — writes, deletes, and admin operations — into an append-only audit log. The v3.0.0-beta.3 release added an audit delete trigger at the SQL layer so that even direct database writes (bypassing the API) leave a forensic trail (Source: v3.0.0-beta.3 release notes).

3. Cache and Webhooks

src/cache.ts wraps the Redis client used for hot-tier caching and reciprocal-rank-boost memoization. Connection lifecycle is owned by the cache module: it opens the client on first use and closes it on shutdown, which is what fixed the test-suite hangs reported in v3.0.0-beta.2 (Source: v3.0.0-beta.2 release notes).

src/webhooks.ts delivers outbound notifications (consolidation complete, memory evicted, etc.) with retry and exponential backoff. Failures are dead-lettered, not swallowed, so operators can replay them.

4. LLM Safety and Data Hygiene

src/llm-safety.ts guards the boundaries between user-supplied content and any prompt sent to an embedding or completion model. It performs three checks: token-length clamping, PII redaction on user-provided fields, and a refusal list for prompt-injection patterns. The module is invoked both on add and on query so embeddings and prompts are both sanitized.

5. Known Issues

The validation bug is a *pre-existing* finding noted during adversarial review of #160 and is therefore out of scope for that PR, but it should be triaged before a production release because it leaks server-side stack traces to clients (Source: issue #162).

6. Operational Runbook Highlights

1. Startup — verify DATABASE_URL, REDIS_URL, and any LLM provider keys are present; the service fails fast via config.ts if they are missing.
2. Migrations — apply canonical SQL migrations (including the RLS policies and audit delete trigger landed in v3.0.0-beta.3) before pointing traffic at a new release.
3. Rollout — v3.0.0-beta.4 is the first release published via npm Trusted Publishing (OIDC); no long-lived token is required in CI (Source: v3.0.0-beta.4 release notes).
4. Sharding — Phase 2 introduces domain partitioning and cold-tier recovery; operators should size the cold tier against the configured MemoryBudget to avoid the new hard-budget back-pressure.
5. Incident triage — correlate the audit log (audit.ts) with webhook delivery failures (webhooks.ts) when debugging dropped events.

7. Release & CI Status

The v3.0.0-beta.2 milestone made the CI pipeline fully green — six jobs across typecheck-lint, integration-tests, and cache-tests on Node 20 and Node 22, all passing (Source: v3.0.0-beta.2 release notes). Subsequent betas have added namespaces, RLS policies, audit triggers, and the memory namespaces feature tracked in issue #16, all without re-introducing the connection-leak class of bugs fixed in beta.2.

Doramagic Pitfall Log

Found 15 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.0.0-beta.3
• User impact: Upgrade or migration may change expected behavior: v3.0.0-beta.3
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.0.0-beta.3. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | https://github.com/salishforge/memforge/releases/tag/v3.0.0-beta.3

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: v3.0.0-beta.4
• User impact: Upgrade or migration may change expected behavior: v3.0.0-beta.4
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.0.0-beta.4. Context: Observed when using node, python
• Evidence: failure_mode_cluster:github_release | https://github.com/salishforge/memforge/releases/tag/v3.0.0-beta.4

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.host_targets | https://github.com/salishforge/memforge

4. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: MemForge v0.1.0-alpha
• User impact: Upgrade or migration may change expected behavior: MemForge v0.1.0-alpha
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: MemForge v0.1.0-alpha. Context: Observed when using docker
• Evidence: failure_mode_cluster:github_release | https://github.com/salishforge/memforge/releases/tag/v0.1.0-alpha

5. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Developers should check this configuration risk before relying on the project: v3.0.0-beta.2 — CI Green, Shared Memory, Full Test Coverage
• User impact: Upgrade or migration may change expected behavior: v3.0.0-beta.2 — CI Green, Shared Memory, Full Test Coverage
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: v3.0.0-beta.2 — CI Green, Shared Memory, Full Test Coverage. Context: Observed when using node, python, docker
• Evidence: failure_mode_cluster:github_release | https://github.com/salishforge/memforge/releases/tag/v3.0.0-beta.2

6. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/salishforge/memforge

7. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/salishforge/memforge/issues/161

8. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/salishforge/memforge/issues/162

9. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/salishforge/memforge

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/salishforge/memforge

11. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/salishforge/memforge

12. Runtime risk: Runtime risk requires verification

• Severity: low
• Finding: Developers should check this performance risk before relying on the project: REST: /entities and /graph return 500 for invalid agent ids; OpenAPI query path missing epistemic param
• User impact: Developers may hit a documented source-backed failure mode: REST: /entities and /graph return 500 for invalid agent ids; OpenAPI query path missing epistemic param
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: REST: /entities and /graph return 500 for invalid agent ids; OpenAPI query path missing epistemic param. Context: Source discussion did not expose a precise runtime context.
• Evidence: failure_mode_cluster:github_issue | https://github.com/salishforge/memforge/issues/162

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using memforge with real data or production workflows.

• benchmark: flat Recall@k curve suggests scoring granularity issue - github / github_issue
• Python SDK: QueryResult dataclass rejects unknown response keys (breaks - github / github_issue
• REST: /entities and /graph return 500 for invalid agent ids; OpenAPI que - github / github_issue
• v3.0.0-beta.4 - github / github_release
• v3.0.0-beta.3 - github / github_release
• v3.0.0-beta.2 — CI Green, Shared Memory, Full Test Coverage - github / github_release
• v2.1.0-alpha — Architecture Complete - github / github_release
• MemForge v0.1.0-alpha - github / github_release
• Configuration risk requires verification - GitHub / issue