# https://github.com/agentscope-ai/ReMe Project Manual Generated at: 2026-06-24 05:02:16 UTC ## Table of Contents - [Overview & Memory-as-File Architecture](#page-1) - [Automatic Memory Pipeline (auto_index, auto_memory, auto_resource, auto_dream, proactive)](#page-2) - [Workspace Operations, CLI, and Agent Integration](#page-3) - [System Components, Configuration & Deployment](#page-4) ## Overview & Memory-as-File Architecture ### Related Pages Related topics: [Automatic Memory Pipeline (auto_index, auto_memory, auto_resource, auto_dream, proactive)](#page-2), [System Components, Configuration & Deployment](#page-4)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) - [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py) - [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py) - [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py) - [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py) - [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py) - [reme/utils/logo_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logo_utils.py) - [reme/components/client/http_client.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/client/http_client.py)
# Overview & Memory-as-File Architecture ## Purpose and Core Philosophy ReMe (Memory Management Kit for Agents) is a self-hosted toolkit that turns raw conversations and external resources into file-based long-term memory for AI agents. Its central design statement is **"Memory as File, File as Memory"**: every memory unit is a Markdown file with frontmatter and `[[wikilinks]]` that humans and agents can read, edit, and search directly. Source: [README.md:1-30](). Four guiding ideas are documented in the README: - **Memory as File** — Markdown with frontmatter and wikilinks forms the canonical storage format. - **Self-evolving knowledge base** — `auto_memory`, `auto_resource`, and `auto_dream` progressively convert raw input into long-term Markdown memories and build wikilink relationships. - **Progressive hybrid search** — Wikilinks, BM25, and embeddings are combined for keyword, semantic, and relationship recall. - **Agent-friendly integration** — `SKILL.md` plus the CLI make it easy for different agents to consume memory. Source: [README.md:14-30](). The intended audiences are personal assistants (e.g., QwenPaw), coding assistants, knowledge QA, and task automation. Source: [README.md:34-45](). ## Workspace Layout and Memory Stages ReMe operates on a `workspace_dir` (renamed from `vault_dir` in v0.4.0.1) laid out as four top-level areas plus a metadata directory. The directory tree is part of the public API — every capability operates on these names. Source: [README.md:55-90](). ```text / ├── metadata/ # Persistent state: indexes, graphs, catalogs ├── session/ # Raw conversations and agent sessions │ ├── dialog/.jsonl │ ├── agentscope/ │ └── claude_code/ ├── resource/ # External raw materials ├── daily/ # Lightly processed memory │ ├── YYYY-MM-DD.md │ └── YYYY-MM-DD/.md, .md, interests.yaml └── digest/ # Long-term memory ├── personal/ ├── procedure/ └── wiki/ ``` `metadata/` holds the persistent indexes that make the workspace searchable and linkable; without it, BM25, embeddings, and wikilink graph queries cannot resolve. The `session/` directory is the raw ingestion layer — note that framework-specific folders (`agentscope/`, `claude_code/`) are versioned subdirectories so multiple framework integrations can coexist without colliding. Source: [README.md:55-90](). ## Automatic Memory Flow ReMe runs four automatic capabilities that progressively turn raw data into long-term knowledge: ```mermaid flowchart LR A[session/ raw dialog] -->|auto_memory| B[daily/ facts & summaries] R[resource/ external files] -->|auto_resource| B B -->|auto_dream cron| C[digest/ long-term nodes] C -->|interests.yaml| P[proactive reminders] I[auto_index watcher] -.-> B I -.-> C ``` Source: [README.md:95-130](). | Capability | Trigger | Effect | |---|---|---| | `auto_index` | Background `index_update_loop` | Scans `daily/`, `digest/`, `resource/`; updates chunk, BM25, embedding, and wikilink indexes. | | `auto_memory` | Agent after-reply hook (or on demand) | Saves raw conversation to `daily//.md`. | | `auto_resource` | Resource file watcher (or on demand) | Creates or updates same-name resource cards in `daily/`. | | `auto_dream` | Scheduled (`dream_cron`) | Extracts long-term units from daily input, integrates into `digest/`, writes `interests.yaml`. | | `proactive` | On demand before agent reply | Reads `interests.yaml`, surfaces topics for the agent to remind the user about. | Source: [README.md:95-130](). The Wikilink layer is the connective tissue between all these directories. `[[X]]` syntax is parsed, rewritten, and validated by `WikilinkHandler`, which also performs graph-aware operations like finding inbound links for `file_delete` and retargeting links after `file_move`. Source: [reme/utils/wikilink_handler.py:1-50](). Targets are taken literally — no implicit `.md`, no short-form basename resolution — and the recommended form is a full path relative to the workspace with extension (e.g., `[[topics/x.md]]`). Anchor and alias survive a rewrite verbatim; image marker (`!`) and Dataview predicate (`pred::` outside the brackets) are not touched. Source: [reme/utils/wikilink_handler.py:1-80](). ## Integration and Operational Notes ReMe exposes both a CLI and an HTTP service. `reme start` defaults to `127.0.0.1:2333`; the startup banner is rendered by `print_logo` and includes runtime config (backend, URL, versions). Source: [README.md:48-54](), [reme/utils/logo_utils.py:1-60](). The unified `Job` interface covers retrieval, read, write, edit, and automatic-memory commands; lower-level commands are reserved for maintenance and integration. Source: [README.md:130-170](). For retrieval, the typical agent loop calls `search` (hybrid vector + BM25 + RRF), `node_search` (digest-level recall by abstraction name/description), `traverse` (wikilink graph walk), and `read` / `read_image` for direct file access. The `search` step additionally expands per-hit context via `expand_links` and `render_expansion_lines`, which group outbound and inbound edges by neighbor and render them as `→ path name=… description=…` blocks. Source: [reme/utils/link_expansion.py:1-80](), [reme/utils/__init__.py:1-30](). The HTTP client is provided as `HttpClient` in `reme/components/client/http_client.py` and yields SSE frames for streaming consumption. Source: [reme/components/client/http_client.py:1-30](). Internal utilities like `execute_stream_task` in `common_utils.py` glue an `asyncio.Queue[StreamChunk]` to a background task so consumers can iterate either as raw chunks or as SSE frames. Source: [reme/utils/common_utils.py:1-50](). For AgentScope integration specifically, agent session state is serialized to JSONL via `AsStateHandler` — line 1 is the `AgentState.summary` header, lines 2+ are `AgentState.context` messages — and is stored under `session/agentscope/`. Source: [reme/utils/agent_state_io.py:1-50](). ### Recurring Community Questions Mapped to This Overview - **Multi-turn + external RAG (Issue #280)**: ReMe is not itself a vector RAG server like RagFlow; it is a workspace + memory layer that an agent can sit *in front of* a RAG system. The hybrid `search` job and the digest layer are where long-term recall happens, not inside a single RAG collection. - **Startup `AssertionError: Invalid type: str` (Issue #25)**: This error typically indicates stale or mismatched configuration values during boot; double-check the `EMBEDDING_*` and `LLM_*` environment variables (the current names; older docs referenced `FLOW_EMBEDDING_*`) and the workspace path resolved by the `service` component. - **Reproducing paper experiments (Issue #200)**: The training-style memory data is built up over multiple `auto_dream` cycles, not in a single pass — each daily consolidation pass refines the digest. - **v0.4.0.1 rename**: `vault_dir` is now `workspace_dir` everywhere, and the package was renamed from `reme` to `reme-ai` (install with `pip install "reme-ai[core]"`). Source: [README.md:48-54](), [v0.4.0.1 release notes](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.1). ## See Also - Memory Subsystems (`auto_memory`, `auto_resource`, `auto_dream`) - Hybrid Search & Wikilink Traversal - CLI and HTTP Service Reference - Agent Integration Patterns (AgentScope / Claude Code) --- ## Automatic Memory Pipeline (auto_index, auto_memory, auto_resource, auto_dream, proactive) ### Related Pages Related topics: [Overview & Memory-as-File Architecture](#page-1), [Workspace Operations, CLI, and Agent Integration](#page-3)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) - [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py) - [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py) - [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py) - [reme/utils/token_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/token_utils.py) - [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py) - [reme/components/client/http_client.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/client/http_client.py) - [docs/zh/auto_memory.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_memory.md) - [docs/zh/auto_resource.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_resource.md) - [docs/zh/auto_dream.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_dream.md) - [docs/zh/proactive.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/proactive.md) - [docs/zh/memory_search.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/memory_search.md)
# Automatic Memory Pipeline (auto_index, auto_memory, auto_resource, auto_dream, proactive) ## Overview The Automatic Memory Pipeline is ReMe's core mechanism for transforming raw conversations and external resources into a structured, searchable, file-based long-term memory store. The pipeline is designed around five cooperating capabilities — `auto_index`, `auto_memory`, `auto_resource`, `auto_dream`, and `proactive` — each running either in the background, on a schedule, or on demand through the same CLI/Service Job interface (`README.md`). The philosophy is **Memory as File, File as Memory**: Markdown files with frontmatter and `[[wikilinks]]` are first-class memory nodes that both users and agents can read, write, and edit directly. The pipeline operates over a `workspace_dir` (formerly `vault_dir`, renamed in v0.4.0.1) organized into four top-level subdirectories: - `session/` — raw conversations and agent sessions - `resource/` — external raw materials grouped by date - `daily/` — lightly processed memory cards - `digest/` — consolidated long-term memory nodes (personal / procedure / wiki) This design was explicitly preferred over sliding-window context truncation (used in systems such as Dify), because once information is moved into ReMe's file-backed memory, it is not "discarded" — it is relocated, indexed, and remains queryable (`README.md` — "Memory System" section). ## Pipeline Architecture ```mermaid flowchart LR subgraph Triggers["Trigger Sources"] A1[Agent reply hook] A2[Resource watcher] A3[dream_cron schedule] A4[On-demand CLI / Service call] end subgraph Pipeline["Automatic Memory Pipeline"] B1[auto_index
background loop] B2[auto_memory
conversation card] B3[auto_resource
resource card] B4[auto_dream
consolidation] B5[proactive
interest surfacing] end subgraph Storage["workspace_dir/"] C1[(session/)] C2[(resource/YYYY-MM-DD/)] C3[(daily/YYYY-MM-DD/)] C4[(digest/personal|procedure|wiki)] C5[(metadata/ indexes & graph)] end A1 --> B2 --> C3 A2 --> B3 --> C3 B2 -.updates.-> B1 B3 -.updates.-> B1 B1 <--> C5 A3 --> B4 --> C4 B4 --> C3 A4 --> B5 B5 --> C3 C5 <--> B1 ``` The diagram captures the data-flow view from the README's "Automatic Memory Flow" section: background watchers maintain indexes, agent hooks trigger conversation memory, scheduled tasks consolidate long-term memory, and proactive reminders are produced on demand. ## Pipeline Stages ### auto_index — Background Index Maintenance `auto_index` is the persistent indexer that runs while the ReMe service is up. It scans the workspace on startup and continuously watches Markdown and JSONL changes inside configured `watch_dirs` (`daily/`, `digest/`, `resource/`). It maintains four parallel indexes: - **Chunk index** — file content broken into retrievable units via the file-chunker components - **BM25 index** — keyword search - **Embedding index** — semantic recall - **Wikilink graph** — relationships inferred from `[[…]]` syntax The wikilink machinery is owned by `WikilinkHandler` in [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py), which is described in that file's docstring as the "single source of truth for `[[…]]` syntax." It handles pure-text parsing (`iter_matches`, `extract_links`) as well as file-graph-aware async operations (`find_inbound`, `retarget_links`). Targets are taken literally — `[[X]]` becomes `target="X"`, with no implicit `.md` expansion or short-form basename search, which keeps graph rewriting deterministic across moves and deletes. The link-expansion utilities used downstream by search live in [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py), where `expand_links` returns structured out/in link data and `render_expansion_lines` formats it as the ` → path name=… description=…` blocks surfaced by `SearchStep`. ### auto_memory — Conversation Memory `auto_memory` is exposed as both an agent **after-reply hook** and an on-demand job (`README.md`, "Automatic Memory Capabilities" table). It takes the conversation `messages` (required) plus an optional `session_id` and `memory_hint`, then: 1. Persists the raw conversation text into `session/dialog/.jsonl` via `AsStateHandler` (see [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py), which serializes `AgentState` to JSONL with the summary on line 1 and the context on subsequent lines). 2. Summarizes long-term valuable information and writes a memory card to `daily//.md`. This addresses the multi-turn conversation concern raised in [issue #280](https://github.com/agentscope-ai/ReMe/issues/280) — ReMe's approach differs from sliding-window truncation because information deemed long-term valuable is *promoted* into `daily/` rather than dropped, and `auto_index` will subsequently pick the new card up for hybrid retrieval. ### auto_resource — Resource Memory `auto_resource` is triggered automatically when files under `resource//` change, and is also callable on demand. The README describes it as reading resource changes and creating/updating same-name daily resource cards under `daily//.md`. Parameters are a `changes` array where each entry typically carries `path`, `file_path`, and a `change` verb, which lets one call cover multiple file-system events without re-traversing the resource tree. ### auto_dream — Long-Term Consolidation `auto_dream` is the only stage with a built-in scheduler (`dream_cron`). For a given `date` (and optional `hint`, `topic_count`, `topic_diversity_days`), it: 1. Scans the daily input for that date in `daily/`. 2. Extracts long-term memory units via the extract step ([reme/steps/evolve/dream/extract.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/steps/evolve/dream/extract.py)). 3. Integrates them into `digest/` under `personal/`, `procedure/`, or `wiki/`. 4. Writes `daily//interests.yaml`, which downstream consumers (such as `proactive`) read. The v0.4.0.0 release notes ([release/v0.4.0.0](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.0)) called out "improve recall workflow and documentation" for `auto_dream`, and v0.4.0.1 ([release/v0.4.0.1](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.1)) renamed `vault_dir` to `workspace_dir` across documentation and examples — both relevant when following older tutorials. ### proactive — Interest Surfacing `proactive` is a read-only, on-demand stage typically called by an upper-level agent before sending a proactive reminder. It reads `interests.yaml` produced by `auto_dream` and returns topics worth attention, leaving the actual reminder decision to the caller (`README.md`). Parameters are `date` and an optional `include_content` flag. Because it is a pure reader, it does not mutate any workspace files. ## Component Reference | Component | Trigger | Required inputs | Writes to | Reads from | |---------------|--------------------------------------|---------------------------------------|----------------------------------------------|-------------------------------------| | `auto_index` | Background `index_update_loop` | Config: `watch_dirs`, `watch_suffixes`| `metadata/` (chunk, BM25, embedding, graph) | `daily/`, `digest/`, `resource/` | | `auto_memory` | Agent after-reply hook / on demand | `messages`; optional `session_id`, `memory_hint` | `session/`, `daily//.md` | incoming agent messages | | `auto_resource` | Resource watcher / on demand | `changes[]` with `path`/`file_path`/`change` | `daily//.md` | `resource//` | | `auto_dream` | `dream_cron` schedule / on demand | `date`, `hint`, `topic_count`, `topic_diversity_days` | `digest/`, `daily//interests.yaml` | `daily//` | | `proactive` | On demand before reminder | `date`, optional `include_content` | — | `daily//interests.yaml` | ## Common Failure Modes and Configuration Notes - **Embedding/chat model must be reachable.** `auto_memory` and `auto_dream` both call the configured LLM/embedding endpoints (e.g., `EMBEDDING_API_KEY`, `LLM_BASE_URL`). Issue [#25](https://github.com/agentscope-ai/ReMe/issues/25) reports an `AssertionError: Invalid type: str` at startup when an OpenAI-compatible endpoint (such as a local `llamacpp` server) returns responses in a slightly different shape than expected; verifying the endpoint's `/v1/models` output and matching the `EMBEDDING_BASE_URL` / `LLM_BASE_URL` to its OpenAI-compatible prefix is the standard mitigation. - **Model substitution must update every consumer.** Issue [#10](https://github.com/agentscope-ai/ReMe/issues/10) reports that changing the documented parameters still resulted in `Qwen-max` being called — a reminder that both the embedding and chat profiles must be re-pointed together; otherwise `auto_index` will still embed against the old embedding model while `auto_memory` / `auto_dream` summarize against the new one. - **`auto_index` is required for retrieval to see new memory cards.** Even if `auto_memory` writes a card to `daily/`, the `search` and `node_search` jobs only return hits once `auto_index` has refreshed the chunk/BM25/embedding/graph indexes (see `reme/utils/wikilink_handler.py` and `reme/utils/link_expansion.py` for the data model those indexes produce). - **Wikilink targets are literal.** From [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py): `[[X]]` resolves to `target="X"` verbatim — no implicit `.md` is appended, no short-form basename search, and no folder-note expansion. Recommended form is a full path relative to workspace with extension (e.g., `[[topics/x.md]]`). This matters for `auto_index` because rewrite operations triggered by file moves/deletions (via `retarget_links` / `find_inbound`) will only update inbound links whose target text matches the moved file's path verbatim. - **Service startup uses HTTP transport.** `reme start` exposes the jobs above over HTTP by default on `127.0.0.1:2333`, with a CLI override (`reme start service.port=8181`). The transport layer uses Server-Sent Events for streamed actions — see `_format_chunk` and `execute_stream_task` in [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py). A separate concern filed in [issue #275](https://github.com/agentscope-ai/ReMe/issues/275) reminds operators to harden the HTTP service (wildcard CORS with credentials, pinned `chromadb` CVE, Neo4j default password) before exposing it beyond localhost. ## Operational Patterns In a typical deployment the pipeline runs with minimal manual intervention: 1. Start the service: `reme start` (or `reme start workspace_dir=/path`). 2. Hook `auto_memory` into the agent's after-reply callback so conversation memory is created without explicit calls. 3. Place external materials in `resource//`; `auto_resource` plus `auto_index` will pick them up and the agent can call `search`, `read`, `traverse`, or `node_search` against the workspace. 4. Let `dream_cron` (or an explicit `auto_dream` call) consolidate yesterday's `daily/` cards into `digest/` and refresh `interests.yaml`. 5. When the agent wants to nudge the user, call `proactive` to read `interests.yaml` and decide whether to surface a topic. Token-aware summarization inside `auto_memory` and `auto_dream` relies on the lightweight byte-based estimator in [reme/utils/token_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/token_utils.py) (`estimate_token_count`, dividing UTF-8 byte length by a configurable divisor). For memory workloads where exact token budgets matter, replace this estimate with a real tokenizer in your deployment profile. ## See Also - [README.md — Memory System and Automatic Memory Flow](https://github.com/agentscope-ai/ReMe/blob/main/README.md) - [docs/zh/auto_memory.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_memory.md) - [docs/zh/auto_resource.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_resource.md) - [docs/zh/auto_dream.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/auto_dream.md) - [docs/zh/proactive.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/proactive.md) - [docs/zh/memory_search.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/memory_search.md) - [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py) - [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py) - [Release v0.4.0.1](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.1) - [Release v0.4.0.0](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.0) - [Issue #280 — Multi-turn conversation + RagFlow architecture](https://github.com/agentscope-ai/ReMe/issues/280) - [Issue #275 — HTTP service hardening (CORS, chromadb CVE, Neo4j password)](https://github.com/agentscope-ai/ReMe/issues/275) --- ## Workspace Operations, CLI, and Agent Integration ### Related Pages Related topics: [Automatic Memory Pipeline (auto_index, auto_memory, auto_resource, auto_dream, proactive)](#page-2), [System Components, Configuration & Deployment](#page-4)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) - [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py) - [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py) - [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py) - [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py) - [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py) - [reme/utils/logo_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logo_utils.py)
# Workspace Operations, CLI, and Agent Integration ## 1. Purpose and Scope ReMe is a file-based memory management toolkit for AI agents. The "Workspace Operations, CLI, and Agent Integration" surface is the contract agents and human operators use to interact with that workspace: the CLI launcher, the HTTP service jobs, the wikilink text/rewrite pipeline that powers retrieval, and the state I/O that bridges an external agent runtime (e.g., AgentScope) with ReMe's on-disk memory. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). The workspace itself is a directory tree (`workspace_dir`, formerly `vault_dir` — renamed in v0.4.0.1) with five top-level regions: `metadata/`, `session/`, `resource/`, `daily/`, and `digest/`. Every operation exposed by the CLI or the Service is ultimately a typed job that reads or mutates files in this tree. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). ## 2. CLI and Service Entry Point The package is distributed as `reme-ai` (renamed from `reme` in v0.4.0.1). Install with `pip install "reme-ai[core]"` and require Python 3.11+. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). Required environment variables (read at startup) are LLM and embedding credentials: ```bash EMBEDDING_API_KEY=sk-xxx EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 LLM_API_KEY=sk-xxx LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 ``` Start the service with `reme start`; the default listen address is `127.0.0.1:2333`. A different port (or other config overrides) can be passed in dotted form, e.g., `reme start service.port=8181`. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). Constants `REME_DEFAULT_HOST` and `REME_DEFAULT_PORT` are defined in [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py). Lower-level entry helpers (`find_reme`, `locate_reme`, `precheck_start`, `cli_find_reme`) are re-exported from [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py), and `print_logo` produces the rainbow ASCII banner from [reme/utils/logo_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logo_utils.py) on startup. ## 3. Workspace Operation Interface ReMe exposes its workspace through a unified CLI / Service Job interface. Agents typically only need retrieval, read, write, edit, and automatic-memory commands; indexing, frontmatter, and file-operation commands are aimed at maintenance, debugging, or advanced integration. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). | Category | Name | Description | Parameters | |---|---|---|---| | System status | `version`, `health_check`, `help` | Package version, health summary, registered jobs | — | | Retrieval / read | `search` | Hybrid retrieval (vector recall, BM25, RRF fusion) | `query`; opt. `limit`, `min_score` | | Retrieval / read | `node_search` | Recall similar digest nodes by abstraction name/description | `query`; opt. `limit` | | Retrieval / read | `traverse` | Traverse the wikilink graph from a path | `path`; opt. `depth`, `direction` | | Retrieval / read | `read`, `read_image` | Read a Markdown file (line range) or image (base64) | `path`; opt. `start_line`, `end_line` | | Index | `reindex` | Clear file-store indexes and rebuild from existing files | `watch_dirs`, `watch_suffixes` | | Daily | `daily_create`, `daily_list`, `daily_reindex` | Create / list / rebuild the day index | `session_id`, `date` | | Metadata | `frontmatter_read`, `frontmatter_update`, `frontmatter_delete` | Read / merge / delete frontmatter keys | `path`, `metadata`, `keys` | | File operation | `stat`, `list` (and `file_delete`, `file_move` behind the scenes) | Path status, recursive listing, graph-aware delete/rename with wikilink retargeting | `path`, etc. | Table adapted from [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). The automatic-memory capabilities (background watchers and on-demand jobs) are likewise exposed as jobs and are the primary integration point for agent runtimes: - `auto_index` — runs in `index_update_loop`; watches `daily/`, `digest/`, `resource/` and refreshes chunk, BM25, embedding, and wikilink graph indexes. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). - `auto_memory` — agent after-reply hook (or on-demand) that saves `messages` into `daily//.md` memory cards. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). - `auto_resource` — automatically triggered by resource watching; turns `resource//` changes into same-name daily resource cards. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). - `auto_dream` — scheduled by `dream_cron`; consolidates `daily/` into `digest/` long-term memory and writes `interests.yaml`. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). - `proactive` — reads `interests.yaml` and exposes topics worth attention to the calling agent. Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). ## 4. Wikilink Mechanics and Link Expansion Wikilinks (`[[target]]`, `[[target#anchor]]`, `[[target|alias]]`, `![[target]]` for images) are the structural backbone of retrieval and graph traversal. All wikilink concerns are owned by `WikilinkHandler` in [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py). Conventionally, targets are taken **literally** — `[[X]]` resolves to `target="X"`, with no implicit `.md`, basename search, or folder-note expansion. Anchor and alias survive a rewrite verbatim; the image marker (`!`) and Dataview predicate (`pred::` outside the brackets) sit outside `[[...]]` and are not touched by a rewrite. The recommended form is a full path relative to the workspace with extension, e.g., `[[topics/x.md]]`. Source: [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py). The handler splits responsibilities cleanly: - Pure-text operations: `iter_matches`, `extract_links`, `scan_and_rewrite`, `validate_src_dst` / `validate_scope` / `within_scope`. - Async, file-graph aware operations: `find_inbound` (used by `file_delete` to surface referencing files) and `retarget_links` (used by `file_move` to point inbound `[[src]]` at the new path). Inbound candidates come from the file graph's reverse index — no filesystem scan. Source: [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py). Link expansion is the view layer that turns outlinks/inlinks into the ` → path name=… description=…` blocks `SearchStep` historically prints. `expand_links` (data layer) returns a structured dict keyed by source path, and `render_expansion_lines` (view layer) formats one path's expansion. Source: [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py). Both are re-exported from [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py). ```mermaid flowchart LR A[Agent / CLI call] --> B[job: search / traverse / read] B --> C[WikilinkHandler.iter_matches] C --> D{File graph} D -- outbound --> E[FileLink edges] D -- inbound --> F[expand_links] F --> G[render_expansion_lines] G --> H[Markdown context block] E --> H ``` ## 5. Agent State I/O and Common Failure Modes `AsStateHandler` in [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py) serializes an `agentscope.state.AgentState` to a JSONL file under `session/`. Line 1 is a header `Msg` carrying `AgentState.summary` plus scalar metadata (e.g., `session_id`, `reply_id`, `cur_iter`); subsequent lines are `AgentState.context` messages, one per line. The factory `AsStateHandler.for_session(directory, session_id)` validates the session id and returns a handler pointing at `/.jsonl`; `load_or_none` is the soft variant used at agent startup, and `delete` is exposed so the workspace can reclaim state files. Source: [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py). Community-reported issues surface several operational pitfalls worth documenting: - **Startup failures with custom LLM endpoints.** Issue #25 reports `AssertionError: Invalid type: str` when configuring `FLOW_EMBEDDING_API_KEY` / `FLOW_LLM_API_KEY` against a local llama.cpp server. Verify that the LLM and embedding base URLs respond in OpenAI-compatible mode and that the requested `model` parameter is honored end-to-end (Issue #10 documents parameters silently being ignored, with the runtime always pointing at `Qwen-max`). Source: [GitHub Issue #25](https://github.com/agentscope-ai/ReMe/issues/25), [GitHub Issue #10](https://github.com/agentscope-ai/ReMe/issues/10). - **Security hygiene on the HTTP service.** Issue #275 flags wildcard CORS combined with credentials, a known `chromadb` CVE on the pinned version, and a Neo4j default-password concern in the adapter. Operators should tighten CORS, pin a patched `chromadb`, and rotate the Neo4j default credential before exposing the service. Source: [GitHub Issue #275](https://github.com/agentscope-ai/ReMe/issues/275). - **Reproducing paper experiments.** Issue #200 asks how many iterations produce the published training memory and whether `run_appworld.py` is sufficient — the answer is iteration-driven via the `auto_dream` job, not a single `run_appworld.py` invocation. Source: [GitHub Issue #200](https://github.com/agentscope-ai/ReMe/issues/200), [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). - **Config-name migrations.** v0.4.0.1 renamed the package (`reme` → `reme-ai`) and the directory option (`vault_dir` → `workspace_dir`); both must be updated in env, CLI overrides, and any embedded scripts. Source: [Release v0.4.0.1](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.1). For general plumbing (hashing via `hash_text`, SSE stream rendering via `execute_stream_task`, the `mock_reme_server` test helper, and the `call_action` / `call_and_check` HTTP wrappers), see [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py) and the re-exports in [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py). ## See Also - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) — overview, directory layout, automatic memory flow, and the canonical workspace operation table. - Project documentation site: [https://reme.agentscope.io/](https://reme.agentscope.io/) - Release notes: [v0.4.0.1](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.1), [v0.4.0.0](https://github.com/agentscope-ai/ReMe/releases/tag/v0.4.0.0) - Related community discussion: [Issue #280 — ReMe in multi-turn + RagFlow architectures](https://github.com/agentscope-ai/ReMe/issues/280) --- ## System Components, Configuration & Deployment ### Related Pages Related topics: [Overview & Memory-as-File Architecture](#page-1), [Workspace Operations, CLI, and Agent Integration](#page-3)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) - [reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py) - [reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py) - [reme/utils/env_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/env_utils.py) - [reme/utils/logger_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logger_utils.py) - [reme/utils/logo_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logo_utils.py) - [reme/utils/service_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/service_utils.py) - [reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py) - [reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py) - [reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py) - [reme/components/base_component.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/base_component.py) - [reme/components/component_registry.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/component_registry.py) - [reme/components/service/base_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/base_service.py) - [reme/components/service/http_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/http_service.py) - [reme/components/service/mcp_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/mcp_service.py) - [reme/components/job/base_job.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/job/base_job.py)
# System Components, Configuration & Deployment ReMe (Memory Management Kit for Agents) is distributed as the `reme-ai` PyPI package and can run as an HTTP service, an MCP service, or a CLI. This page covers the component model, the configuration surface, and the deployment workflow that ties them together. As of release v0.4.0.1, the project ships under the `reme-ai` package name (renamed from `reme`), and the on-disk root is consistently called `workspace_dir` (renamed from `vault_dir`) ([README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md)). ## Component Architecture ReMe follows a strict CLI / Service / Application / Job / Step / Component layering. Every runnable unit extends the base `Component` class and is registered in a shared `ComponentRegistry` ([reme/components/base_component.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/base_component.py), [reme/components/component_registry.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/component_registry.py)). This means configuration, dependency wiring, and lifecycle are uniform across retrieval jobs, auto-memory pipelines, and the background watcher loop. There are two principal service surfaces: | Service | Source | Purpose | |---------|--------|---------| | `BaseService` | [reme/components/service/base_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/base_service.py) | Defines lifecycle, register/unregister hooks, and shared config | | `HttpService` | [reme/components/service/http_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/http_service.py) | FastAPI-style HTTP transport that exposes the Job interface | | `McpService` | [reme/components/service/mcp_service.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/service/mcp_service.py) | Model Context Protocol transport for agent tool calling | Jobs are long-running or on-demand operations (e.g. `auto_memory`, `auto_resource`, `auto_dream`, `search`, `read`, `reindex`) registered through `BaseJob` ([reme/components/job/base_job.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/components/job/base_job.py)) and exposed uniformly by both services. ## Configuration Model ReMe reads configuration from three sources, in increasing priority: defaults, a `.env` file, and CLI overrides of the form `reme start service.port=8181` ([README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md)). The required environment variables are: ```bash EMBEDDING_API_KEY=sk-xxx EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 LLM_API_KEY=sk-xxx LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 ``` Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). `load_env` and `parse_env_file` centralize reading these values ([reme/utils/env_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/env_utils.py)), while `ApplicationConfig` aggregates runtime settings such as the `workspace_dir` and the service binding ([reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py)). The default service address is `127.0.0.1:2333`, defined by `REME_DEFAULT_HOST` and `REME_DEFAULT_PORT` constants ([reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py)). ```mermaid flowchart LR A[Defaults] --> D[ApplicationConfig] B[.env file] --> C[load_env / parse_env_file] --> D E[CLI overrides
service.port=8181] --> D D --> S[HttpService / McpService] D --> J[BaseJob registry] ``` ## Utility and Helper Layer The `reme/utils` package collects cross-cutting helpers used by components, services, and CLI ([reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py)): - **Streaming and HTTP**: `execute_stream_task` and `mock_reme_server` provide Server-Sent Events plumbing and an in-process mock used by tests; `call_action` and `call_and_check` are the standard ways components invoke each other ([reme/utils/common_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/common_utils.py)). - **Logging**: A single `get_logger` factory configures a project-wide logger ([reme/utils/logger_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logger_utils.py)). - **Banner**: `print_logo` renders a rainbow ASCII logo with backend, URL, and version metadata at startup ([reme/utils/logo_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/logo_utils.py)). - **Service lifecycle**: `find_reme`, `locate_reme`, `precheck_start`, and `cli_find_reme` resolve an existing ReMe process and validate the workspace before startup ([reme/utils/service_utils.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/service_utils.py)). - **Numeric helpers**: `cosine_similarity`, `batch_cosine_similarity`, and `estimate_token_count` support vector ranking and budget checks ([reme/utils/__init__.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/__init__.py)). - **AgentState I/O**: `AsStateHandler` serializes/deserializes an `AgentState` to a JSONL file under `session/`, with a header line carrying the summary and one Msg per subsequent line ([reme/utils/agent_state_io.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/agent_state_io.py)). - **Wikilink handling**: `WikilinkHandler` is the single source of truth for `[[...]]` parsing, validation, and graph-aware rewrites ([reme/utils/wikilink_handler.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/wikilink_handler.py)). - **Link expansion**: `expand_links` and `render_expansion_lines` build the per-hit context shown by `SearchStep` ([reme/utils/link_expansion.py](https://github.com/agentscope-ai/ReMe/blob/main/reme/utils/link_expansion.py)). ## Deployment & Operations The standard deployment path is the CLI: ```bash pip install "reme-ai[core]" # or: pip install -e ".[core]" reme start # binds 127.0.0.1:2333 reme start service.port=8181 # override via dotted key ``` Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). The first run creates the workspace layout that backs the memory model: ```text / ├── metadata/ # indexes, graphs, catalogs ├── session/ # raw conversations (jsonl) ├── resource/ # external raw materials ├── daily/ # lightly processed memory cards └── digest/ # long-term personal / procedure / wiki memory ``` Source: [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md). Background `auto_index` watches Markdown/JSONL changes in `daily/`, `digest/`, and `resource/`, and rebuilds chunk, BM25, embedding, and wikilink-graph indexes on demand via the `reindex` job. `auto_memory` runs as an agent hook to persist each turn, and `auto_dream` consolidates a day's input into long-term digest memory on a `dream_cron` schedule ([README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md)). ### Common Failure Modes Community-reported issues map to this layer as follows: - **Startup error `AssertionError: Invalid type: str`** (issue #25) — typically caused by an unsupported model identifier or a missing `EMBEDDING_API_KEY` / `LLM_API_KEY` pair; verify the env block above before launching. - **Model substitution ignored** (issue #10) — overrides must use the dotted CLI syntax (e.g. `reme start llm.model=...`); raw flag-style arguments are not parsed. - **Security defaults** (issue #275) — review `HttpService` CORS, the Neo4j adapter default credentials, and the pinned `chromadb` version before exposing the service on a network. - **Package rename** (v0.4.0.1) — the import name is now `reme_ai`; older `import reme` snippets will fail until updated. ## See Also - [README.md](https://github.com/agentscope-ai/ReMe/blob/main/README.md) — full project overview and CLI reference - [docs/zh/quick_start.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/quick_start.md) — extended quick start - [docs/zh/framework.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/framework.md) — CLI/Service/Application/Job/Step/Component layering details - [docs/zh/memory_search.md](https://github.com/agentscope-ai/ReMe/blob/main/docs/zh/memory_search.md) — `search` and `auto_index` semantics --- --- ## Pitfall Log Project: agentscope-ai/ReMe Summary: Found 8 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Security or permission risk - Security or permission risk requires verification. ## 1. Security or permission risk - Security or permission risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/agentscope-ai/ReMe/issues/280 ## 2. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/agentscope-ai/ReMe ## 3. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/agentscope-ai/ReMe ## 4. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/agentscope-ai/ReMe ## 5. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/agentscope-ai/ReMe ## 6. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/agentscope-ai/ReMe/issues/275 ## 7. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/agentscope-ai/ReMe ## 8. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/agentscope-ai/ReMe