# https://github.com/ahwurm/localharness Project Manual

Generated at: 2026-07-20 12:27:33 UTC

## Table of Contents

- [Overview and System Architecture](#page-1)
- [Memory, Learning, and Recall Pipeline](#page-2)
- [Agent Loop, Tooling, and Permissions](#page-3)
- [Deployment, Benchmarking, and Operations](#page-4)

<a id='page-1'></a>

## Overview and System Architecture

### Related Pages

Related topics: [Memory, Learning, and Recall Pipeline](#page-2), [Agent Loop, Tooling, and Permissions](#page-3), [Deployment, Benchmarking, and Operations](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/ahwurm/localharness/blob/main/README.md)
- [src/localharness/__init__.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/__init__.py)
- [src/localharness/core/bus.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/core/bus.py)
- [src/localharness/core/events.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/core/events.py)
- [src/localharness/core/reduce.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/core/reduce.py)
- [src/localharness/core/types.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/core/types.py)
- [src/localharness/agent/loop.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/loop.py)
- [src/localharness/memory/discovery.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/discovery.py)
- [src/localharness/server.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/server.py)
</details>

# Overview and System Architecture

## Purpose and Scope

`localharness` is an interactive REPL harness for running a locally-served LLM agent with first-class memory and tool use. It bundles a model server manager (vLLM via `docker` or a local process), a single-flight inference client, an event-sourced agent loop, and a durable memory pipeline under one CLI. The goal is a dogfoodable environment where every turn's actions, ambient memory injections, and discovery outcomes are observable as events, so behaviour can be audited and replayed. Source: [README.md:1-40]()

The system is deliberately narrow in scope: it is not a framework for building agents from scratch, but a fixed-shape harness that wires a model to a tool surface, a memory store, and a turn-level event log. Subsystems communicate through an event bus rather than direct calls, which keeps the agent loop composable and testable. Source: [src/localharness/core/bus.py:1-40]()

## High-Level Component Map

The runtime decomposes into five cooperating layers, each with a clear boundary:

- **CLI / REPL front-end** — owns the interactive prompt, banner, and slash-commands such as `/model`. It dispatches to slash-command handlers and forwards free-form input to the agent loop. Source: [README.md:41-80]()
- **Server manager** — supervises the vLLM server (`launch: docker` or `launch: local`), exposes named `local_models` entries with per-checkpoint `vllm serve` args, and supports in-REPL full model swap. Source: [src/localharness/server.py:1-80]()
- **Inference client** — a single-flight client that serialises stream requests behind one semaphore, reused by both the agent and the type-anytime router. Source: [src/localharness/agent/loop.py:1-60]()
- **Agent loop** — drives each turn: classify input, plan, call tools, render, finalize, emit `TurnCompleted`. Source: [src/localharness/agent/loop.py:60-260]()
- **Memory pipeline** — mines semantic atoms, maintains an active set with unique-key constraints, runs algorithmic tag discovery, and writes per-turn ambient injection traces. Source: [src/localharness/memory/discovery.py:1-120]()

```mermaid
flowchart LR
    User --> REPL
    REPL --> AgentLoop
    AgentLoop --> InferenceClient
    InferenceClient --> Server[vLLM Server]
    AgentLoop --> Bus[Event Bus]
    Bus --> Reducer
    AgentLoop --> Memory[Memory Pipeline]
    Memory --> Discovery
    Memory --> Store[(Durable Store)]
```

## Event-Sourced Core

The core package implements a minimal, deterministic event log. Events are plain dataclasses declared in `core/types.py` and emitted through the bus in `core/bus.py`. The reducer in `core/reduce.py` folds events into snapshots that downstream subsystems (memory, traces, UI) consume without coupling to the loop. Source: [src/localharness/core/events.py:1-60]()

Notable lifecycle events include `TurnStarted`, `TurnCompleted`, `MemoryInjected`, and `ServerSwap`, all of which are first-class on the bus. Turn finalization is what guarantees a `TurnCompleted` event reaches the bus; if the REPL exits before finalization runs, that turn is left without bookkeeping, which is observable in the trace store. Source: [src/localharness/agent/loop.py:200-260]()

The event bus is intentionally thin: synchronous publish for in-process subscribers, with no durability guarantees beyond what each subscriber chooses to persist. This keeps the agent loop's hot path off any heavy I/O while still allowing the memory and trace pipelines to record faithfully. Source: [src/localharness/core/bus.py:40-90]()

## Agent Loop and Turn Lifecycle

Each REPL turn passes through a fixed sequence inside `agent/loop.py`: classification, planning, optional tool calls, answer rendering, and finalization. A type-anytime router can re-classify mid-turn; its tier-2 path reuses the agent's own LLM client and therefore serialises behind the same single-flight gate as streaming replies. Source: [src/localharness/agent/loop.py:60-180]()

The act-guard is a single-shot nudge that fires when the model emits no tool calls: it asks the model to reply `CONFIRMED` if the answer is already final, so the harness can avoid one more round trip. The completion summary helper treats a bare `CONFIRMED` as a sentinel and falls back to the last assistant content in `session.messages`. Source: [src/localharness/agent/loop.py:220-260]()

Tool execution is mediated by a write gate that captures novelty facts; the gate's novelty check is session-scoped, which means the same "first successful use" of a tool can re-fire across days if the session state is reset while the durable store still holds the fact. Source: [src/localharness/memory/discovery.py:120-200]()

## Memory and Discovery

Memory is organised as semantic atoms in the `sem/` namespace. Atoms have a stable active key, a payload, and metadata; the durable store enforces uniqueness on the active key, so two facts sharing a key upsert into one. However, mined atoms that contradict each other do not collide on this key by default — both can remain active simultaneously because each mined fact receives a unique key. Source: [src/localharness/memory/discovery.py:60-160]()

Algorithmic tag discovery proposes candidate tags with `status="proposed"` and `origin="discovered"`. Each pass unconditionally bumps `last_accrual_ts` whenever an unchanged cluster still matches a candidate, which can defeat staleness pruning and leave candidates in limbo. Promotion to named status requires explicit evidence; in long-running stores, candidates have remained at `proposed` for extended periods. Source: [src/localharness/memory/discovery.py:260-310]()

Ambient injection writes a per-turn trace row listing the atom ids that were placed on the turn's memory shelf. The trace write is gated on the injected-id list's truthiness, so turns whose shelf is empty produce no trace row — the "nothing was injected" event is therefore unrecorded and skews per-turn coverage metrics. Source: [src/localharness/memory/discovery.py:200-260]()

## Server Management

The server manager owns the vLLM process lifecycle. In `launch: docker` mode it shells out to `docker run --rm`; in `launch: local` mode it spawns `vllm serve` directly. `server.local_models` registers named checkpoints, each with its own `vllm serve` arguments, so model-specific flags (quantization, MoE backend) never leak to sibling models. The `/model` slash command lists registry names alongside live and HF-cache models; selecting one restarts the managed server with that checkpoint. Source: [src/localharness/server.py:80-180]()

A known sharp edge is that the harness does not `wait()` on the `docker run --rm` client. If the container crashes during startup (for instance, a quantization flag incompatible with the checkpoint), the client exits and the harness is left with a zombie process under the REPL, defeating any dead-process fail-fast the managed server is supposed to provide. Source: [src/localharness/server.py:180-240]()

## Related Community Discussions

- Server lifecycle fragility with `docker run --rm` clients: see issue #99.
- Act-guard sentinel behaviour and completion-summary fallback: see issue #98.
- Memory contradiction handling between `sem/` atoms: see issue #95.
- Ambient injection trace gaps for empty shelves: see issue #96.
- Discovery candidates stuck at `proposed`: see issue #90.
- Write-gate novelty re-firing across sessions: see issue #89.

---

<a id='page-2'></a>

## Memory, Learning, and Recall Pipeline

### Related Pages

Related topics: [Overview and System Architecture](#page-1), [Agent Loop, Tooling, and Permissions](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/localharness/memory/sqlite.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/sqlite.py)
- [src/localharness/memory/gate.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/gate.py)
- [src/localharness/memory/predictive_gate.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/predictive_gate.py)
- [src/localharness/memory/predictive_write_gate.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/predictive_write_gate.py)
- [src/localharness/memory/discovery.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/discovery.py)
- [src/localharness/memory/consolidation.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/memory/consolidation.py)
</details>

# Memory, Learning, and Recall Pipeline

## Purpose and Scope

The `localharness/memory/` package owns the agent's persistent knowledge: deciding **what to remember**, **how to store it**, **how to surface it on later turns**, and **when to consolidate**. The pipeline runs alongside the conversational REPL — each turn potentially produces candidate facts, but only some pass through the write gates, get mined into semantic atoms, are consolidated, and become eligible for re-injection on future turns.

The package is split by responsibility rather than by data shape:

| Module | Role |
|---|---|
| `sqlite.py` | Durable atom store with active-key uniqueness |
| `gate.py` | Synchronous write gate for newly observed facts |
| `predictive_gate.py` | Pre-turn prediction of likely useful recall |
| `predictive_write_gate.py` | Predict-before-write novelty scoring |
| `discovery.py` | Algorithmic tag and cluster mining |
| `consolidation.py` | Turn-end and background consolidation passes |

## Durable Storage

`sqlite.py` is the single source of truth. Semantic atoms live in an active table guarded by a unique-key constraint, plus a history of superseded rows so that deletions are non-destructive. Every mined fact carries an `active_key` — a stable identifier intended to collide on duplicates so the store can upsert rather than accumulate redundant rows. Source: [src/localharness/memory/sqlite.py]()

A known limitation follows directly from the key design (issue #95): because each mined semantic atom mints its own key, **directly contradictory facts can coexist as active simultaneously**. The active-key uniqueness constraint only catches exact duplicates, not semantic opposites, so the store can hold e.g. an active "vLLM server listens on port 8000" alongside an active "...port 8081". There is no built-in contradiction resolution. Source: [src/localharness/memory/sqlite.py:1-200]()

## Write Gates and Novelty Capture

Three gate modules sit in front of the store:

- `gate.py` is the synchronous, per-observation gate. It decides whether a freshly observed fact (a tool result, a user statement, an error) is worth recording at all.
- `predictive_gate.py` runs before a turn and predicts which recall slots will actually be used, so the agent does not waste context on irrelevant atoms.
- `predictive_write_gate.py` does the analogous work on the *write* side — scoring whether a new candidate is novel enough to be worth persisting given what is already active. Source: [src/localharness/memory/predictive_write_gate.py]()

The write-gate novelty capture is currently scoped to session state rather than the durable store (issue #89). The "First successful use of tool X observed" fact can therefore re-fire on later days; the unique-active-key constraint then upserts silently, bumping `updated_at` instead of recognizing the fact as already known. Source: [src/localharness/memory/gate.py]()

## Discovery: Tags and Clusters

`discovery.py` runs algorithmic passes over the active atom set to mine **tag candidates** and **atom clusters**. Mined tags are stamped `status: proposed`, `origin: discovered`, and are intended to be model-NAMED later and promoted on accumulating evidence. Source: [src/localharness/memory/discovery.py]()

Two operational defects are documented in community issues:

1. **Permanent limbo** (issue #94). Every discovery pass unconditionally bumps `last_accrual_ts` on a proposed candidate whenever the same unchanged atom cluster still matches it (`discovery.py` ~:286–290), even when zero new members or sitter events have arrived. This defeats any staleness-pruning logic that depends on the timestamp gap. Source: [src/localharness/memory/discovery.py:286-290]()
2. **Naming and promotion unobserved** (issue #90). In real stores, dozens of `proposed` candidates created across many days still carry the placeholder status — the model-naming step and the evidence-promotion step are not actually running on them.

## Consolidation

`consolidation.py` runs at turn end and, separately, as a background micro-pass. Its job is to fold raw observations into semantic atoms, retire stale rows, and (in principle) move `proposed` tags toward `named`/`active`. The turn-end micro-pass is invoked from the agent loop after each completed turn, which is why an immediate REPL exit (Ctrl+C right after the answer renders) can skip a turn's memory treatment and produce no `TurnCompleted` event (issue #93). Source: [src/localharness/memory/consolidation.py]()

## Recall: Ambient Injection

Recall is implemented as **ambient injection**: before each turn, a "memory shelf" of relevant atoms is built from the active store and prepended to the context transparently. The injected-id list is also written to a per-turn trace so coverage can be measured downstream.

A defect in this trace (issue #96): the trace write is gated on the injected-id list's truthiness, so turns whose shelf is empty (typical for the first few turns of a fresh store, before any fact exists) produce no trace row at all. The "nothing was injected" event is therefore unrecorded, which skews per-turn coverage statistics and per-store recall evaluations. Source: [src/localharness/agent/loop.py]()

## End-to-End Flow

```mermaid
flowchart LR
  A[Tool result / user input] --> B[gate.py<br/>write gate]
  B -->|novel| C[predictive_write_gate.py<br/>novelty scoring]
  C -->|persist| D[sqlite.py<br/>active atom]
  D --> E[consolidation.py<br/>turn-end pass]
  E --> F[discovery.py<br/>cluster and tag mining]
  F -->|proposed| D
  D --> G[predictive_gate.py<br/>per-turn recall]
  G --> H[ambient shelf<br/>to next-turn context]
  H --> I[trace row<br/>injected-id list]
```

The pipeline is best understood as a loop: every turn produces both **write traffic** (observations → gates → store → consolidation → discovery) and **read traffic** (store → predictive gate → ambient shelf → context). The open issues above cluster around three failure modes — contradiction without resolution (#95), discovery candidates stuck in `proposed` (#90, #94), and gaps in turn-end and trace instrumentation (#93, #96) — which together explain most of the visible "memory drift" users report.

---

<a id='page-3'></a>

## Agent Loop, Tooling, and Permissions

### Related Pages

Related topics: [Overview and System Architecture](#page-1), [Deployment, Benchmarking, and Operations](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/localharness/agent/loop.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/loop.py)
- [src/localharness/agent/context.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/context.py)
- [src/localharness/agent/permissions.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/permissions.py)
- [src/localharness/agent/subagent.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/subagent.py)
- [src/localharness/orchestrator/router.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/orchestrator/router.py)
- [src/localharness/orchestrator/cards.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/orchestrator/cards.py)
</details>

# Agent Loop, Tooling, and Permissions

## Purpose and Scope

The agent layer in localharness is the runtime that turns a user prompt into one or more model turns, executes tool calls, gates their side effects, and persists the resulting state into the conversation and memory subsystems. It is the connective tissue between the REPL/CLI entry points (`src/localharness/cli`, `src/localharness/repl`) and the orchestration primitives that route, classify, and record work. Three concerns live here:

- The **agent loop** (`agent/loop.py`) — turn construction, completion summarization, and end-of-turn bookkeeping.
- **Tooling** — the routing/classification layer (`orchestrator/router.py`), the typed tool-card registry (`orchestrator/cards.py`), and sub-agent dispatch (`agent/subagent.py`).
- **Permissions** — the write-gate and act-guard machinery (`agent/permissions.py`) that decides whether a tool's intended effect is allowed, deferred, or rewritten before it ever reaches a backend.

These three concerns are intentionally separated so that model-side reasoning, capability discovery, and safety policy can evolve independently.

## Agent Loop Lifecycle

A single user turn traverses roughly the following pipeline inside `agent/loop.py`:

1. **Context assembly** — `agent/context.py` builds the message list, ambient memory shelf, and any injected tool outputs from prior turns. Source: [src/localharness/agent/context.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/context.py)
2. **Streamed inference** — the loop calls the model's chat completion API through a single-flight inference gate. The gate serializes overlapping calls so that the tier-2 input classifier and the agent's own stream do not contend on the same connection. Source: [src/localharness/agent/loop.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/loop.py)
3. **Tool-call dispatch** — any `tool_calls` in the assistant message are forwarded to the orchestrator router, which selects a card and invokes it.
4. **Permission gating** — every call is checked against `agent/permissions.py`; permitted calls run, deferred ones return a structured refusal that the model can react to, and forbidden ones are dropped with a trace row.
5. **Completion summarization** — `_format_completion_summary` condenses the turn's tool results and assistant content for storage. A bare `CONFIRMED` sentinel triggers a fallback to `_last_assistant_content(session.messages)`. Source: [src/localharness/agent/loop.py:246-256]()
6. **Turn finalization** — `TurnCompleted` is emitted, end-of-turn micro-passes run, and the memory pipeline treats the turn.

```mermaid
flowchart LR
    U[User input] --> C[Context assembly\nagent/context.py]
    C --> R[Orchestrator router\norchestrator/router.py]
    R -->|classify| T2[Tier-2 classify]
    R -->|pick| K[Tool card\norchestrator/cards.py]
    K --> P{Permissions\nagent/permissions.py}
    P -->|allow| X[Execute tool]
    P -->|defer/refuse| M[Refusal message]
    X --> S[Streamed reply]
    M --> S
    S --> F[Finalize + summarize\nagent/loop.py]
    F --> Mem[Memory pipeline]
```

## Tooling: Router, Cards, and Sub-agents

The orchestrator exposes tools through a **card registry** rather than raw function dispatch. A card (`orchestrator/cards.py`) wraps a callable with a JSON Schema describing its arguments, a permission class, an expected latency bucket, and a human-readable description. The router (`orchestrator/router.py`) maps a model's `tool_calls` payload to a card by name, then re-validates arguments against the schema before invocation. Source: [src/localharness/orchestrator/router.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/orchestrator/router.py)

**Sub-agents** (`agent/subagent.py`) are themselves represented as cards: spawning a focused agent is just another tool call. The parent loop treats the child's completion as a single tool result, which keeps the parent's context window bounded and makes nesting explicit in the trace.

The tier-2 classifier is used to decide, mid-turn, whether the user prompt needs a tool at all. When classification serializes behind the agent's own stream and the stream holds the single-flight inference semaphore for its full duration, the classify call can be starved — and any failure path is silent rather than surfaced to the model. This is tracked in community issue #92.

## Permissions and Gating

`agent/permissions.py` is the policy boundary between *what the model asked for* and *what the harness will actually do*. It evaluates each proposed tool call against:

- The tool's declared permission class (read, write-with-approval, write-autonomous).
- A session-scoped allow/deny ledger maintained by the REPL (e.g. the write-gate novelty capture).
- A configurable list of side-effect categories (shell exec, network egress, file mutation).

A novel observation in the write-gate is meant to fire **once per tool per store**, but the current check is session-scoped rather than durable-store-scoped, so the same first-use fact re-fires on later days and is silently upserted by the unique-active-key constraint. This is community issue #89. Source: [src/localharness/agent/permissions.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/agent/permissions.py)

The **act-guard** is a separate nudge: when the model answers with zero tool calls on a turn that the harness expected to act, it injects a one-shot reminder to reply `CONFIRMED` if the task genuinely needs no tools. A downstream bug (#98) causes that sentinel to occasionally replace the user-visible answer rather than pass the original reply through, because `_format_completion_summary` treats `CONFIRMED` as a sentinel and falls back to `_last_assistant_content(session.messages)` — and `session.messages` is the full multi-turn conversation, so a stale prior turn can resurface.

## Community Notes

- **#89 Write-gate novelty capture** re-claims first-use facts daily because the novelty check is session-scoped, not durable. Source: [src/localharness/agent/permissions.py]()
- **#91 / #98 CONFIRMED sentinel echo** — `_format_completion_summary` interprets a bare `CONFIRMED` reply as a sentinel; the fallback path can resurface a prior turn's content. Source: [src/localharness/agent/loop.py:246-256]()
- **#92 Mid-turn tier-2 classification** is starved by the single-flight inference gate and fails silently. Source: [src/localharness/orchestrator/router.py]()
- **#93 Skipped finalization** when the REPL exits immediately after an answer, before `TurnCompleted` fires. Source: [src/localharness/agent/loop.py]()
- **#96 Empty-shelf turns** produce no ambient-injection trace row because the trace write is gated on a truthy injected-id list. Source: [src/localharness/agent/context.py]()

These issues cluster around the boundary between streaming inference and post-turn bookkeeping — the seam where single-flight gating, permission evaluation, and summarization all meet.

---

<a id='page-4'></a>

## Deployment, Benchmarking, and Operations

### Related Pages

Related topics: [Overview and System Architecture](#page-1), [Memory, Learning, and Recall Pipeline](#page-2), [Agent Loop, Tooling, and Permissions](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/localharness/provider/client.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/provider/client.py)
- [src/localharness/provider/server.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/provider/server.py)
- [src/localharness/provider/detector.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/provider/detector.py)
- [src/localharness/provider/fn_call.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/provider/fn_call.py)
- [src/localharness/provider/refarch.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/provider/refarch.py)
- [src/localharness/cli/model_cmd.py](https://github.com/ahwurm/localharness/blob/main/src/localharness/cli/model_cmd.py)
</details>

# Deployment, Benchmarking, and Operations

This page documents how the harness provisions, swaps, and operates the inference backend it depends on, and where benchmarking plugs into that lifecycle. The deployment/operations surface is concentrated in `src/localharness/provider/server.py` and the matching CLI in `src/localharness/cli/model_cmd.py`, while the HTTP, detection, and protocol layers live in `src/localharness/provider/client.py`, `src/localharness/provider/detector.py`, `src/localharness/provider/fn_call.py`, and `src/localharness/provider/refarch.py`.

## Managed Server Lifecycle

The harness can run in front of an externally-managed inference endpoint or it can own the server lifecycle itself. When the operator opts into a managed server, the `server.launch` setting selects between a native/local `vllm serve` process and a `launch: docker` mode that invokes `docker run --rm` with the configured image and flags. Source: [src/localharness/provider/server.py:1-80]().

In `docker` mode the harness spawns the `docker run` client, monitors the container, and exposes `server_pid()` to the REPL. The current implementation has a known gap: when the vLLM container crashes during startup (for example because a quantization flag does not match the checkpoint), the `docker run --rm` client exits but the harness does not `wait()` on it, so the dead client lingers as a zombie under the REPL process. This defeats the managed-server fail-fast signal — the REPL sees the server as healthy because the client process still exists — and is tracked in [#99](https://github.com/ahwurm/localharness/issues/99). Operators working around this must manually reap the `docker` client before retrying.

## Local Model Registry and In-REPL Model Swap

As of v0.9.23, `server.local_models` is a registry of named local checkpoints, each carrying its own `vllm serve` argument set. The motivation is isolation: a MoE-only backend flag or a checkpoint-specific quantization flag should never leak to a sibling model in the registry. Source: [src/localharness/provider/server.py:80-160]().

The `/model` REPL command lists three categories side by side:

- Registry entries from `server.local_models`
- Live models currently served by the backend
- HF-cache checkpoints discovered on disk

Picking any entry restarts the managed server with that checkpoint's argument set. The `/model` picker is implemented in `src/localharness/cli/model_cmd.py`, and the restart orchestration lives in `src/localharness/provider/server.py`. Because a swap is a full restart, operators should size per-model `vllm serve` arguments defensively (memory limits, `--max-model-len`, `--gpu-memory-utilization`) so a smaller checkpoint cannot silently OOM on a larger allocation that the previous model was using.

## Benchmarking and Capability Detection

The harness auto-detects the served model and its capabilities before each session. Detection logic in `src/localharness/provider/detector.py` queries the OpenAI-compatible `/v1/models` endpoint exposed by `src/localharness/provider/client.py` and matches the response against the reference architecture table in `src/localharness/provider/refarch.py` to resolve the correct function-calling protocol from `src/localharness/provider/fn_call.py`.

| File | Role in the stack |
| --- | --- |
| `provider/client.py` | OpenAI-compatible HTTP client for chat completions and model probing |
| `provider/detector.py` | Capability and model-id detection on startup and after swaps |
| `provider/fn_call.py` | Function-calling protocol implementation |
| `provider/refarch.py` | Reference architectures mapping model IDs to supported features |
| `cli/model_cmd.py` | `/model` REPL command and picker UI |

Benchmark runs reuse the same HTTP client, so they exercise the same protocol path as REPL traffic and benefit from the same capability detection. When the model is swapped in-REPL, detection re-runs — a benchmark that needs tool use will only see the tools that the freshly-loaded checkpoint actually supports, never stale capability flags from the previous model.

## Operational Concerns

Three operational details recur across recent releases and community reports:

- **Version banner.** The CLI banner reads its version from installed distribution metadata, not from `localharness.__version__`. On an editable install (the standard dev/dogfood workflow), pulling new source updates the code but not the dist-info, so the banner reports the version from the last `uv sync`. See [#97](https://github.com/ahwurm/localharness/issues/97). The workaround is `uv pip install -e .` or any equivalent metadata-refreshing step.
- **Restart isolation.** `/model` performs a full server restart; previously-set `--max-model-len`, KV-cache sizing, and GPU memory fractions must be re-asserted via the registry entry for the new checkpoint.
- **Zombie cleanup.** Until the `wait()` fix in [#99](https://github.com/ahwurm/localharness/issues/99) lands, an operator who sees a dead managed server should manually reap the `docker run --rm` client (e.g. via `pidof docker` plus `kill`) before retrying — otherwise the harness will mis-report the server as healthy.

The combination of `server.launch`, `server.local_models`, and the `/model` picker is intended to make deployment, swap, and benchmarking a single uniform workflow: declare a checkpoint once, pick it from the REPL, and the harness takes care of restart, redetection, and re-binding the function-calling protocol against the new backend.

---

<!-- evidence_pipeline_checked: true -->
<!-- evidence_injected: true -->

---

## Pitfall Log

Project: ahwurm/localharness

Summary: Found 39 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

## 1. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this installation risk before relying on the project: Version banner reads stale install metadata on editable installs — shows old version after every source update
- User impact: Developers may fail before the first successful local run: Version banner reads stale install metadata on editable installs — shows old version after every source update
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/97

## 2. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/90

## 3. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/94

## 4. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/93

## 5. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/97

## 6. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/99

## 7. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.host_targets | https://github.com/ahwurm/localharness

## 8. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: Zombie docker-run client defeats the managed server's dead-process fail-fast
- User impact: Developers may misconfigure credentials, environment, or host setup: Zombie docker-run client defeats the managed server's dead-process fail-fast
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/99

## 9. Capability evidence risk - Capability evidence risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/ahwurm/localharness

## 10. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this runtime risk before relying on the project: Mid-turn tier-2 input classification is starved by the single-flight inference gate, and its failure path is silent
- User impact: Developers may hit a documented source-backed failure mode: Mid-turn tier-2 input classification is starved by the single-flight inference gate, and its failure path is silent
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/92

## 11. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/84

## 12. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/95

## 13. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/85

## 14. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/92

## 15. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/88

## 16. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/91

## 17. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/96

## 18. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/86

## 19. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/89

## 20. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/87

## 21. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/ahwurm/localharness/issues/98

## 22. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/ahwurm/localharness

## 23. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/ahwurm/localharness

## 24. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://github.com/ahwurm/localharness

## 25. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Agent loop accepts a tool-less reply that announces further work as the turn's final answer
- User impact: Developers may hit a documented source-backed failure mode: Agent loop accepts a tool-less reply that announces further work as the turn's final answer
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/84

## 26. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Contradictory sem/ atoms coexist as active — mined facts never supersede on contradiction
- User impact: Developers may hit a documented source-backed failure mode: Contradictory sem/ atoms coexist as active — mined facts never supersede on contradiction
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/95

## 27. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Discovery candidate tags never progress past 'proposed' — naming/promotion unobserved for 6+ days
- User impact: Developers may hit a documented source-backed failure mode: Discovery candidate tags never progress past 'proposed' — naming/promotion unobserved for 6+ days
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/90

## 28. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Messages typed into the input box leave no permanent record in the terminal transcript
- User impact: Developers may hit a documented source-backed failure mode: Messages typed into the input box leave no permanent record in the terminal transcript
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/85

## 29. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Mint-time classification can assign BOTH L1 buckets to one atom
- User impact: Developers may hit a documented source-backed failure mode: Mint-time classification can assign BOTH L1 buckets to one atom
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/88

## 30. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Turn summary fallback can resurface a PRIOR turn's reply when the model answers with only the confirmation sentinel
- User impact: Developers may hit a documented source-backed failure mode: Turn summary fallback can resurface a PRIOR turn's reply when the model answers with only the confirmation sentinel
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/91

## 31. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Working indicator renders in the input box's bottom border, crowding queue count and decision flash
- User impact: Developers may hit a documented source-backed failure mode: Working indicator renders in the input box's bottom border, crowding queue count and decision flash
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/86

## 32. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Write-gate novelty capture re-claims 'first successful use' of the same tool across days
- User impact: Developers may hit a documented source-backed failure mode: Write-gate novelty capture re-claims 'first successful use' of the same tool across days
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/89

## 33. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: remember()-saved memories are never tagged at write time; backfill runs so late in consolidation that user activity starves it
- User impact: Developers may hit a documented source-backed failure mode: remember()-saved memories are never tagged at write time; backfill runs so late in consolidation that user activity starves it
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/87

## 34. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this conceptual risk before relying on the project: Act-guard sentinel echo replaces the user-visible answer with the CONFIRMED meta-line
- User impact: Developers may hit a documented source-backed failure mode: Act-guard sentinel echo replaces the user-visible answer with the CONFIRMED meta-line
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/98

## 35. Runtime risk - Runtime risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this performance risk before relying on the project: Discovery candidates can enter permanent limbo: evidence re-accrual on unchanged clusters defeats staleness pruning
- User impact: Developers may hit a documented source-backed failure mode: Discovery candidates can enter permanent limbo: evidence re-accrual on unchanged clusters defeats staleness pruning
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/94

## 36. Runtime risk - Runtime risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this performance risk before relying on the project: Exiting the REPL immediately after an answer skips that turn's end-of-turn bookkeeping
- User impact: Developers may hit a documented source-backed failure mode: Exiting the REPL immediately after an answer skips that turn's end-of-turn bookkeeping
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/93

## 37. Runtime risk - Runtime risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this performance risk before relying on the project: Turns with an empty memory shelf are never recorded in injection traces
- User impact: Developers may hit a documented source-backed failure mode: Turns with an empty memory shelf are never recorded in injection traces
- Evidence: failure_mode_cluster:github_issue | https://github.com/ahwurm/localharness/issues/96

## 38. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/ahwurm/localharness

## 39. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/ahwurm/localharness

<!-- canonical_name: ahwurm/localharness; human_manual_source: deepwiki_human_wiki -->