# https://github.com/OpenHands/ToM-SWE Project Manual Generated at: 2026-06-15 01:16:59 UTC ## Table of Contents - [Project Overview and System Architecture](#page-1) - [Core ToM Components, Memory, and Generation](#page-2) - [Stateful SWE Benchmark, Session Washing, and Analytics](#page-3) - [OpenHands Integration, REST API, Visualization, and Deployment](#page-4) ## Project Overview and System Architecture ### Related Pages Related topics: [Core ToM Components, Memory, and Generation](#page-2), [OpenHands Integration, REST API, Visualization, and Deployment](#page-4)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) - [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) - [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) - [tom_swe/prompts/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/__init__.py) - [stateful_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/__init__.py) - [stateful_swe/session_washer.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/session_washer.py) - [stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py) - [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py) - [utils/aggregate_user_behavior.py](https://github.com/OpenHands/ToM-SWE/blob/main/utils/aggregate_user_behavior.py)
# Project Overview and System Architecture ## Purpose and Scope ToM-SWE is a Theory of Mind (ToM) package that augments Software Engineering agents with personalized user understanding and adaptive behavior. It provides consultation capabilities that help agents infer user intent, preferences, and working styles, producing improved task performance for downstream code-generation systems. The package is designed to integrate seamlessly with OpenHands and other SWE agent frameworks, exposing a Python API and a forthcoming FastAPI server for remote consumption. Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) At its core, the system targets three capabilities requested by the community: - Proposing improved, personalized instructions for a user request. - Suggesting next actions an agent should take on behalf of a user. - Sending messages to a ToM-aware agent through a thin messaging layer. Source: [issue #17](https://github.com/OpenHands/ToM-SWE/issues/17) The project description recorded in the repository confirms the same scope: ToM-SWE delivers "personalized instruction improvement and user behavior analysis" backed by a three-tier memory system that escalates from raw cleaned sessions, through session analyses, into user profiles. Source: [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) ## Repository Layout The repository is organized into three top-level Python packages and a set of utility modules. The following summary is derived from the package docstrings and the project overview file. | Package | Role | Entry-point files | |---|---|---| | `tom_swe/` | Core library: ToM agent, RAG agent, memory, prompts, structured generation. | `tom_agent.py`, `tom_module.py`, `rag_module.py`, `prompts/manager.py`, `memory/` | | `stateful_swe/` | Benchmark generation: washes sessions, generates profiles, builds evaluation datasets. | `session_washer.py`, `profile_generator.py`, `dataset_builder.py`, `clarity_eval.py` | | `utils/` | Reusable helpers: LLM client wrappers, data utilities, aggregated user analysis. | `llm_client.py`, `data_utils.py`, `aggregate_user_behavior.py` | Source: [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md), [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py), [stateful_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/__init__.py) The package's public surface is re-exported from `tom_swe/__init__.py`, which exposes the data classes (`UserAnalysis`, `SessionAnalysis`, `UserMessageAnalysis`, `UserProfile`, `SWEAgentSuggestion`), the agent factories (`create_tom_agent`, `create_rag_agent`), and the conversation processor (`CleanSessionStore`). Logging is auto-configured on import if `tom_swe.logging_config` is available. Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) A centralized prompt registry backs every workflow. It uses Jinja2 templates and is accessed through `PromptManager`, `get_prompt_manager`, and the `render_prompt` helper. Source: [tom_swe/prompts/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/__init__.py) ## System Architecture The system follows a layered pipeline that turns raw conversation history into actionable agent behavior. The data flow architecture and design patterns are summarized in the project documentation. ```mermaid flowchart LR A[Raw Sessions
data/sessions] --> B[combine_events_to_trajectory.py] B --> C[Trajectory Data] C --> D[user_interaction_analysis.py] C --> E[tom_module / tom_agent] D --> F[User Analysis
data/user_analysis] E --> G[User Model
data/user_model] E --> H[RAG Module
Document chunks] H --> I[ToM Agent
suggest_next_actions
propose_instructions] I --> J[SWE Agent
e.g. OpenHands] J --> K[Personalized Response] ``` Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) Key architectural choices documented for contributors: - **Three-tier memory**: Cleaned sessions feed session analyses, which in turn build user profiles. The memory layer lives under `tom_swe/memory/` and includes the `CleanSessionStore` exposed in the package root. Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) - **RAG retrieval**: User messages are chunked together with surrounding context and metadata (session title, repository, branch, trigger) before being embedded for retrieval. Source: [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py) - **LLM integration**: A `litellm` proxy fronts the model layer, with `litellm_proxy/claude-sonnet-4-20250514` as the primary model and Claude 3.5 Sonnet / Claude Haiku configured as fallbacks for tests. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - **Structured outputs**: Pydantic models are used throughout the pipeline to validate LLM responses. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) ## Stateful SWE Benchmark and Evaluation The `stateful_swe/` package exists to construct and evaluate datasets that probe whether an agent can recover user preferences from conversational history. Three responsibilities are visible in the source files: 1. **Session washing**: `session_washer.py` rewrites raw transcripts so that each user turn carries natural-language preference signals — for example, varying phrasing for verbose vs. concise users, or injecting "ask all questions upfront" for users who dislike interruptions during work. Source: [stateful_swe/session_washer.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/session_washer.py) 2. **Profile generation**: `profile_generator.py` enumerates 15 diverse concrete profiles that cover the interaction combinations of concise/verbose × upfront/ongoing × short/long response styles. Source: [stateful_swe/profile_generator.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/profile_generator.py) 3. **Clarity evaluation**: `multi_model_clarity_eval.py` compares multiple LLMs on a clarity-classification task, tracking accuracy, false positives/negatives, and per-consultation cost in USD. Source: [stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py) The benchmark dataset itself is hosted externally and can be retrieved from the linked Hugging Face repository referenced in the project's issue tracker. Source: [issue #29](https://github.com/OpenHands/ToM-SWE/issues/29) ## Design Patterns and Release Trajectory Several recurring patterns are documented for contributors: - **Async processing**: `asyncio` is used extensively for concurrent LLM calls and data processing. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - **Multi-level caching**: The web trajectory viewer caches metadata and individual conversations to keep interactive exploration responsive. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - **Graceful degradation**: The analyzers fall back to rule-based logic whenever the LLM endpoint is unavailable. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - **Modular analysis**: Statistical analysis and psychological modeling are deliberately separated into distinct modules. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) The release history on GitHub shows the project moving from 1.0.0 (initial release) through typing updates (1.0.1) and package cleanup (1.0.2) to the current 1.0.3 "size optimization" build. Source: [Releases](https://github.com/OpenHands/ToM-SWE/releases). The README also announces an October 2025 beta testing program and provides a one-line `uvx` installer for the OpenHands integration. Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) ## Community Context Several open community threads directly shape the architecture described above: - **OpenHands integration** (#11): The team is exploring a hosted MCP/API server so OpenHands can call into the ToM agent externally; a FastAPI server exposing `suggest_next_actions`, `propose_instructions`, and message delivery is the agreed direction. Source: [issue #11](https://github.com/OpenHands/ToM-SWE/issues/11), [issue #17](https://github.com/OpenHands/ToM-SWE/issues/17) - **Repo refactor** (#14): The package was renamed from `tom_module/` to `tom_swe/`, and `rag_agent.py` was renamed to `rag_module.py` to consolidate the public surface. Source: [issue #14](https://github.com/OpenHands/ToM-SWE/issues/14) - **CI hygiene** (#10, #8): Contributors have requested pre-commit hooks and a mypy + pytest GitHub workflow modeled after the sotopia-lab configuration. Source: [issue #10](https://github.com/OpenHands/ToM-SWE/issues/10), [issue #8](https://github.com/OpenHands/ToM-SWE/issues/8) - **Documentation drift** (#30): A reported 404 in the README's link to `tom_swe/prompts/registry.py` reflects the earlier rename to `tom_swe/prompts/manager.py`; the public re-exports in `tom_swe/prompts/__init__.py` are the source of truth. Source: [issue #30](https://github.com/OpenHands/ToM-SWE/issues/30) ## See Also - [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) - [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) - [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) - [tom_swe/prompts/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/__init__.py) - [stateful_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/__init__.py) - [Issue #11 – Connect with OpenHands](https://github.com/OpenHands/ToM-SWE/issues/11) - [Issue #14 – Create ToM Agent](https://github.com/OpenHands/ToM-SWE/issues/14) - [Issue #17 – RESTful API for ToM agent](https://github.com/OpenHands/ToM-SWE/issues/17) - [Issue #29 – Stateful SWE benchmark data access](https://github.com/OpenHands/ToM-SWE/issues/29) --- ## Core ToM Components, Memory, and Generation ### Related Pages Related topics: [Project Overview and System Architecture](#page-1), [Stateful SWE Benchmark, Session Washing, and Analytics](#page-3), [OpenHands Integration, REST API, Visualization, and Deployment](#page-4)
Related Source Files The following source files were used to generate this page: - [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) - [tom_swe/tom_agent.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_agent.py) - [tom_swe/tom_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_module.py) - [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py) - [tom_swe/memory/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/memory/__init__.py) - [tom_swe/memory/store.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/memory/store.py) - [tom_swe/memory/local.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/memory/local.py) - [tom_swe/memory/conversation_processor.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/memory/conversation_processor.py) - [tom_swe/prompts/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/__init__.py) - [tom_swe/prompts/manager.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/manager.py) - [tom_swe/generation/dataclass.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/generation/dataclass.py) - [stateful_swe/profile_generator.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/profile_generator.py) - [stateful_swe/session_washer.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/session_washer.py) - [stateful_swe/clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/clarity_eval.py) - [stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py)
# Core ToM Components, Memory, and Generation ## Overview and Purpose The `tom_swe` package implements a **Theory of Mind (ToM)** layer that augments Software Engineering (SWE) agents with personalized user understanding. It exposes three orthogonal subsystems that work together: (1) a **core ToM engine** that produces psychological analyses, (2) a **three-tier memory system** that persists cleaned sessions, session analyses, and user profiles, and (3) a **generation layer** that structures LLM input/output and centralizes prompt templates. The package self-describes as "LLM-powered Theory of Mind analysis for user behavior prediction," and its top-level `__all__` list re-exports every public symbol so downstream agents can import them directly from `tom_swe` ([tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py)). The package also auto-configures logging on import, gracefully falling back when `logging_config` is absent. ```mermaid flowchart LR A[User Sessions] --> B[conversation_processor
CleanSessionStore] B --> C[tom_module
ToMAnalyzer] C --> D[memory/store
UserModelStore] D --> E[tom_agent
ToMAgent] E --> F[OpenHands SWE Agent] G[prompts/manager
PromptManager] --> C G --> E H[generation/dataclass
Pydantic Models] --> C H --> E I[rag_module
RAGAgent] --> E ``` ## Core ToM Components ### `ToMAnalyzer` and `ToMAgent` `ToMAnalyzer` (in `tom_module.py`) is the analytical core, while `ToMAgent` (in `tom_agent.py`) is the orchestration entry point that downstream applications actually instantiate. The package ships factory helpers `create_tom_agent()` and `create_rag_agent()` so callers do not need to know the internal config class names ([tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py)). `ToMAgent` exposes two consulting operations that are central to the OpenHands integration and to the planned REST API ([Issue #17](https://github.com/OpenHands/ToM-SWE/issues/17)): - `suggest_next_actions(user_id, context)` — predicts what the user is likely to do next. - `propose_instructions(user_id, raw_request)` — rewrites a vague request into personalized guidance. The `stateful_swe/clarity_eval.py` script demonstrates a third consulting behavior, classifying problem statements as clear or unclear and emitting a `confidence_score` ([stateful_swe/clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/clarity_eval.py)). The `EvalResult` dataclass used there is reused by `stateful_swe/multi_model_clarity_eval.py` to benchmark multiple LLM backends (e.g., `GPT-5-Nano`) on this same task ([stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py)). ### `RAGAgent` and `VectorDB` `rag_module.py` provides retrieval-augmented generation over per-user conversation corpora. Key exports include `RAGAgent`, `VectorDB`, `Document`, `RetrievalResult`, `ChunkingConfig`, `load_processed_data`, and the `create_rag_agent` factory ([tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py)). Internally, `RAGAgent` builds markdown-formatted chunks like: ``` ## User Message ### Context ### Metadata **Chunk ID**: `` **Session**: **Repository**: <selected_repository> **Branch**: <selected_branch> **Trigger**: <trigger> ``` Source: [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py). The surrounding-context extractor and chunking config are configurable, allowing the agent to be tuned for context window size and recall. ## Memory System The memory layer is intentionally split into a small, abstract interface and a file-based backend so it can later be swapped for a database or remote store ([Issue #14](https://github.com/OpenHands/ToM-SWE/issues/14)). ### Public Interface `tom_swe/memory/__init__.py` re-exports `UserModelStore`, `FileStore`, `LocalFileStore`, and a `load_user_model` helper ([tom_swe/memory/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/memory/__init__.py)). The top-level `tom_swe/__init__.py` additionally exposes `CleanSessionStore` from `conversation_processor` ([tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py)). The evaluation scripts in `stateful_swe/` use `LocalFileStore` directly to persist evaluation state ([stateful_swe/clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/clarity_eval.py)). ### Three-Tier Pipeline The memory pipeline (Cleaned sessions → Session analyses → User profiles) is the data backbone the ToM agent reasons over. The package explicitly advertises this in its core features list ([README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md)). | Tier | Producing Component | Consumer | Purpose | |------|--------------------|----------|---------| | Cleaned sessions | `CleanSessionStore` (`memory/conversation_processor.py`) | Session analyzers | Normalized, re-washable session records | | Session analyses | `ToMAnalyzer` / generation pipeline | Profile builder | Per-session psychological summaries | | User profiles | `stateful_swe/profile_generator.py` (`generate_15_profiles`) | `ToMAgent` lookups | Stable, queryable user models | The `profile_generator.py` script materializes 15 diverse profiles covering the full grid of `concise`/`verbose` × `upfront`/`ongoing` × `short_response`/`verbose_response` combinations and uses real coding preferences as seeds ([stateful_swe/profile_generator.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/profile_generator.py)). These generated profiles are useful both for evaluation and for the `session_washer` to inject realistic preference signals back into existing sessions ([stateful_swe/session_washer.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/session_washer.py)). ## Generation and Prompts ### Structured Output with Pydantic `tom_swe/generation/dataclass.py` defines the structured payload types the LLM must produce. The top-level `__init__.py` re-exports `SWEAgentSuggestion`, `UserAnalysis`, `SessionAnalysis`, `UserMessageAnalysis`, and `UserProfile` ([tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py)). Downstream modules use these models both as parser targets and as in-memory representations that are persisted through the memory store. ### Jinja2 Prompt Management `prompts/manager.py` replaces the older `string.format()` approach (the `registry.py` path referenced in the README is no longer valid — see [Issue #30](https://github.com/OpenHands/ToM-SWE/issues/30)) with a Jinja2-based `PromptManager`. It loads templates from a `templates/` directory next to the module, enables `trim_blocks` and `lstrip_blocks` for clean rendering, and registers a custom `length` filter ([tom_swe/prompts/manager.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/manager.py)). Public helpers exposed via `prompts/__init__.py` are `PromptManager`, `get_prompt_manager`, and `render_prompt` ([tom_swe/prompts/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/__init__.py)). ### Integration with the Agent Loop In a typical consultation, `ToMAgent` calls `PromptManager.render(...)` to assemble a system prompt, sends the rendered prompt plus the user's `UserProfile` (or freshly built session context) to the LLM, and parses the response into the dataclasses from `generation/dataclass.py`. The result is then written back through `UserModelStore` so subsequent calls benefit from the updated analysis. This loop is exactly what the planned FastAPI surface ([Issue #17](https://github.com/OpenHands/ToM-SWE/issues/17)) and the OpenHands MCP integration ([Issue #11](https://github.com/OpenHands/ToM-SWE/issues/11)) are expected to wrap. ## See Also - [OpenHands integration and the `TomCodeActAgent` configuration](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) - [Stateful SWE benchmark and dataset access](https://huggingface.co/datasets/cmu-lti/stateful) — discussed in [Issue #29](https://github.com/OpenHands/ToM-SWE/issues/29) - [Prompt management: `prompts/manager.py`](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/prompts/manager.py) — note the dead link reported in [Issue #30](https://github.com/OpenHands/ToM-SWE/issues/30) - [Multi-model clarity evaluation harness](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py) --- <a id='page-3'></a> ## Stateful SWE Benchmark, Session Washing, and Analytics ### Related Pages Related topics: [Project Overview and System Architecture](#page-1), [Core ToM Components, Memory, and Generation](#page-2) <details> <summary>Related Source Files</summary> The following source files were used to generate this page: - [stateful_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/__init__.py) - [stateful_swe/session_washer.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/session_washer.py) - [stateful_swe/profile_generator.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/profile_generator.py) - [stateful_swe/sleeptime_process.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/sleeptime_process.py) - [stateful_swe/visualize_model_results.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/visualize_model_results.py) - [stateful_swe/run_model_comparison.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/run_model_comparison.py) </details> # Stateful SWE Benchmark, Session Washing, and Analytics ## Purpose and Scope The `stateful_swe` package is a benchmark and dataset-generation toolkit that evaluates Theory of Mind (ToM) agents' ability to learn user preferences from conversation history and adapt their behavior accordingly. As stated in the package init, "This module generates datasets for testing Theory of Mind (ToM) agents' ability to learn user preferences from conversation history and adapt their behavior" (Source: [stateful_swe/__init__.py:1-7]()). The package exposes a single `__version__ = "0.1.0"` constant and acts as a self-contained subsystem alongside the core `tom_swe` agent package. The benchmark is composed of three major subsystems: 1. **Profile Generation** — defines realistic power-user personas with interaction and coding preferences. 2. **Session Washing** — rewrites raw user session transcripts to inject those preferences as observable signals. 3. **Analytics & Evaluation** — replays the modified sessions through the `ToMAgent` "sleeptime" compute path and benchmarks downstream clarity/suggestion quality across multiple LLMs. The resulting dataset is publicly available on HuggingFace at `cmu-lti/stateful`, as confirmed in closed community issue [#29](https://github.com/OpenHands/ToM-SWE/issues/29). ## Architecture and Data Flow The pipeline is intentionally linear: a raw session is first "washed" so preference signals become visible, then the washed session is fed back into the ToM agent's offline (`sleeptime_compute`) pass, and finally the resulting outputs are compared across foundation models. ```mermaid flowchart LR A[Profile Generator<br/>15 concrete profiles] --> B[Session Washer<br/>inject preferences] C[Raw user sessions] --> B B --> D[Washed session JSONL] D --> E[Sleeptime Process<br/>tom_agent.sleeptime_compute] E --> F[Per-user model files<br/>data/stateful_swe/usermodeling/<id>/] F --> G[Multi-Model Clarity Eval<br/>+ Visualize] G --> H[summary_table.html<br/>+ error analysis] ``` ## Profile Generation `stateful_swe/profile_generator.py` defines a `ConcreteUserProfile` class with three orthogonal interaction dimensions and a long list of concrete coding preferences (Source: [stateful_swe/profile_generator.py:24-44]()). A typed `ProfileConfig` (a `TypedDict`) captures the full configuration in one structure, while the class itself exposes `to_dict()` and `get_roleplay_prompt()` for downstream use. | Dimension | Allowed Values | Meaning | |---|---|---| | `verbosity` | `concise` / `verbose` | How much detail the user wants from the agent | | `question_timing` | `upfront` / `ongoing` | Whether clarifications happen before or during work | | `response_style` | `short_response` / `verbose_response` | How the user themselves writes messages | The `generate_15_profiles()` method enumerates combinations of these axes (Source: [stateful_swe/profile_generator.py:52-72]()), starting from the eight `(verbosity × question_timing × response_style)` base combinations. The roleplay prompt is then assembled by mapping each axis to a paragraph of natural-language instructions, which can be fed to an LLM during benchmark rollout to simulate that user type. ## Session Washing `stateful_swe/session_washer.py` is the core transformation step. It loads a raw user session plus a target `ConcreteUserProfile`, then uses an LLM to rewrite the user messages so the profile's preferences are "naturally" visible in the transcript. The module's default LLM is `DEFAULT_LLM_MODEL = "gpt-5-mini-2025-08-07"` (Source: [stateful_swe/session_washer.py:30-40]()), and it gracefully degrades when `tom_swe.generation` is not importable by logging a warning and disabling LLM-based processing (Source: [stateful_swe/session_washer.py:21-28]()). The transformation vocabulary is encoded in a `MessageAction` enum with values `KEEP_UNCHANGED`, `REPLACE_SENTENCE`, `APPEND_PREFERENCE`, and `DELETE_CONTENT` (Source: [stateful_swe/session_washer.py:42-50]()). Each per-message change is wrapped in a Pydantic `MessageModification` model (Source: [stateful_swe/session_washer.py:52-58]()), giving the pipeline a structured, machine-readable diff that can be inspected or reverted. The embedded prompt template explicitly enumerates how to handle each preference class. For `response_style`, it gives concrete before/after examples, e.g. "Could you please help me implement a user authentication system with proper security?" → "Implement user auth with security." for `SHORT_RESPONSE` (Source: [stateful_swe/session_washer.py:9-20]()). For coding preferences, the prompt instructs the rewriter to inject them only "when relevant to the request" and to do so naturally — e.g. "Use TypeScript for this" rather than as a tag (Source: [stateful_swe/session_washer.py:30-42]()). ## Sleeptime Processing After washing, `stateful_swe/sleeptime_process.py` replays every washed file through the ToM agent's `sleeptime_compute` function to produce a stable, cached user model per user. The script's module docstring states that outputs are written to `data/stateful_swe/usermodeling/{user_id}/` (Source: [stateful_swe/sleeptime_process.py:1-10]()), making each persona's user-model artifacts individually addressable for downstream evaluation. Concretely, the script: - Loads washed JSON files via `load_washed_session_file()` (Source: [stateful_swe/sleeptime_process.py:25-34]()). Note that the file is read as a single JSON object (not JSONL), one record per file. - Transforms each session into the format expected by `sleeptime_compute` via `extract_sessions_for_sleeptime()` (Source: [stateful_swe/sleeptime_process.py:36-50]()). - Instantiates a `ToMAgent` with a `ToMAgentConfig` and a `LocalFileStore` (Source: [stateful_swe/sleeptime_process.py:6-12]()), and iterates with a `tqdm` progress bar. This step is what makes the benchmark "stateful": a downstream SWE agent evaluation can later read the cached model and observe the ToM agent's predictions on the same persona it was trained against. ## Analytics: Multi-Model Clarity Evaluation Two companion scripts support comparing foundation models on the benchmark's downstream clarity-assessment task. `stateful_swe/multi_model_clarity_eval.py` defines a `MultiModelClarityEvaluator`, and `stateful_swe/run_model_comparison.py` provides a CLI that supports `--all`, `--models <names>`, `--analyze-only`, and `--samples N` (Source: [stateful_swe/run_model_comparison.py:1-13]()). `get_available_models()` enumerates the supported backends: `gpt-5-nano`, `gpt-5-mini`, `gpt-5`, `claude-3.5-sonnet`, `claude-4` (Source: [stateful_swe/run_model_comparison.py:17-19]()), and the model-name filter does a tolerant substring match that strips hyphens and dots. `stateful_swe/visualize_model_results.py` consumes the JSON report produced by the evaluator and renders it to `summary_table.html` along with a per-model error analysis. Errors are bucketed into "false positives" (model suggested questions for clear statements) and "false negatives" (model did not suggest questions for unclear ones), with rates computed against the totals per `statement_type` (Source: [stateful_swe/visualize_model_results.py:1-40]()). The HTML summary also includes a "Cost Effectiveness Rank: Accuracy per dollar spent" column, giving benchmark runners a way to compare models on accuracy-per-cost rather than accuracy alone. ## Common Failure Modes - **Dead README link** — community issue [#30](https://github.com/OpenHands/ToM-SWE/issues/30) flags that `tom_swe/prompts/registry.py` is referenced from the README but returns 404. Many session-washing prompts are loaded by name and will fail at runtime if the registry is missing. - **LLM dependency missing** — `session_washer.py` will log a warning and skip LLM-based washing if `tom_swe.generation` is not installed (Source: [stateful_swe/session_washer.py:21-28]()); the resulting output will then lack preference signals and silently corrupt the benchmark. - **Model name normalization** — the runner's substring matching is lenient (e.g. `claude-4` matches `claude4`), which can produce surprising groupings or even match multiple model records. - **Single-record JSON assumption** — `sleeptime_process.py` reads each washed file as a single JSON object, not JSONL; feeding it concatenated JSONL will raise a parse error (Source: [stateful_swe/sleeptime_process.py:25-34]()). ## See Also - [Theory of Mind Agent and Memory Tiers](./tom-swe-agent.md) - [Prompt Registry and Templates](./prompts-registry.md) - [OpenHands / TomCodeActAgent Integration](./tom-codeact-agent.md) --- <a id='page-4'></a> ## OpenHands Integration, REST API, Visualization, and Deployment ### Related Pages Related topics: [Project Overview and System Architecture](#page-1), [Core ToM Components, Memory, and Generation](#page-2) <details> <summary>Related Source Files</summary> The following source files were used to generate this page: - [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) - [RELEASE.md](https://github.com/OpenHands/ToM-SWE/blob/main/RELEASE.md) - [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) - [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) - [tom_swe/tom_agent.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_agent.py) - [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py) - [stateful_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/__init__.py) - [stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py) - [stateful_swe/visualize_model_results.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/visualize_model_results.py) - [utils/aggregate_user_behavior.py](https://github.com/OpenHands/ToM-SWE/blob/main/utils/aggregate_user_behavior.py) </details> # OpenHands Integration, REST API, Visualization, and Deployment ## Overview ToM-SWE delivers its user-modeling capabilities through three complementary surfaces: (1) an embeddable Python package consumed by the OpenHands SWE-agent runtime, (2) a REST/MCP surface planned by the maintainers for remote consultation, and (3) a suite of CLI and HTML visualization tools that inspect the resulting user models. This page documents each surface strictly against the source files, the README, and the open community issues that define its roadmap. Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) Source: [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) ## OpenHands Agent Integration The README describes ToM-SWE as "a Theory of Mind package designed to enhance Software Engineering agents with personalized user understanding and adaptive behavior" that "integrates seamlessly with OpenHands and other SWE agent frameworks, providing consultation capabilities". Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) The integration entry point is `TomCodeActAgent`, which AGENTS.md identifies as the wrapper that "propose[s] improved instructions/next actions for individual users" by composing the RAG module with the ToM module. Beta-testing instructions install it directly from the OpenHands fork: ```bash pip install uv uvx --python 3.12 \ --from git+https://github.com/XuhuiZhou/OpenHands@feature/tom-codeact-agent \ openhands ``` Source: [AGENTS.md](https://github.com/OpenHands/ToM-SWE/blob/main/AGENTS.md) Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) Internally, `ToMAgent` (exposed via `tom_swe/__init__.py`) coordinates three subsystems: | Subsystem | Module | Role | |---|---|---| | ToM reasoning | `tom_swe.tom_agent.ToMAgent` | Orchestrates `ToMAnalyzer` + RAG context to answer consultation queries | | Profile / memory | `tom_swe.memory.conversation_processor` | Three-tier memory (cleaned sessions → session analyses → user profiles) | | Generation | `tom_swe.generation` | LLM calls and Pydantic-validated structured output | Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) Source: [tom_swe/tom_agent.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_agent.py) The RAG side (chunking, vector DB, retrieval) is implemented in `tom_swe/rag_module.py` and exposed through factory helpers `create_rag_agent` and `create_tom_agent`. Source: [tom_swe/rag_module.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/rag_module.py) Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) ## REST API and MCP Surface (Planned) A REST server has not yet landed under `tom_swe/`. The contract is specified by community issue #17 ("write a restful API for tom agent"), which requires a FastAPI server exposing three endpoints: - `POST /suggest_next_actions` — wraps `ToMAgent.suggest_next_actions` - `POST /propose_instructions` — wraps `ToMAgent.propose_instructions` - `POST /send_message` — forwards a user message into `ToMAgent` Source: GitHub issue [#17](https://github.com/OpenHands/ToM-SWE/issues/17) Companion issue #11 ("Connect with Openhands") recommends hosting the same capabilities behind an MCP server so OpenHands can consume them remotely, since OpenHands speaks MCP natively rather than importing Python directly. As of the current `main` branch, neither the FastAPI nor the MCP server has been merged; the endpoints above should be treated as a target contract rather than an existing API. Source: GitHub issue [#11](https://github.com/OpenHands/ToM-SWE/issues/11) The building blocks the eventual API would compose already exist: `ToMAgent` supports bidirectional consultation (user queries and agent questions), and the structured suggestion type `SWEAgentSuggestion` is part of the public `tom_swe` export list. Source: [tom_swe/tom_agent.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_agent.py) Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) ## Visualization Tooling The repository ships several standalone visualizers that operate on persisted pipeline outputs rather than the live API: - **Trajectory viewers** — `web_trajectory_viewer.py` (Flask, web) and `display_trajectory.py` (Rich, terminal) render processed session trajectories. - **Behavior visualizer** — `user_behavior_visualizer.py` produces statistical charts of user interaction patterns. - **Complexity visualizer** — `complexity_visualizer.py` renders code-complexity analyses. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) For the Stateful SWE benchmark, `stateful_swe/visualize_model_results.py` consumes the report produced by `MultiModelClarityEvaluator` and writes `summary_table.html`. The page header explicitly ranks models by "Cost Effectiveness Rank: Accuracy per dollar spent (higher is better)". The same file splits error patterns into false positives (ToM suggests questions for clear statements) and false negatives (ToM fails to flag unclear statements), giving operators a deployability signal beyond raw accuracy. Source: [stateful_swe/visualize_model_results.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/visualize_model_results.py) Source: [stateful_swe/multi_model_clarity_eval.py](https://github.com/OpenHands/ToM-SWE/blob/main/stateful_swe/multi_model_clarity_eval.py) For per-user auditability, `utils/aggregate_user_behavior.py` emits a Markdown summary containing system-prompt guidelines, communication best practices, workflow optimization, frustration prevention, and proactive-assistance recommendations for a single user model. Source: [utils/aggregate_user_behavior.py](https://github.com/OpenHands/ToM-SWE/blob/main/utils/aggregate_user_behavior.py) ## Deployment and Release `tom-swe` is distributed as a standard Python package on PyPI. The README documents three install paths: ```bash pip install tom-swe # from PyPI pip install uv && uv sync # from a local clone ``` Source: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) `RELEASE.md` codifies a Trusted-Publishing workflow: a GitHub Actions job authenticates with PyPI via OIDC using project `tom-swe`, owner `All-Hands-AI`, repository `ToM-SWE`, workflow file `publish-to-pypi.yml`, and protected environment `pypi`. A TestPyPI rehearsal path is documented for pre-production verification. Source: [RELEASE.md](https://github.com/OpenHands/ToM-SWE/blob/main/RELEASE.md) Runtime configuration is environment-driven: `LITELLM_API_KEY` and `LITELLM_BASE_URL` (or equivalent) are loaded via `.env`, and absent these the pipeline falls back to rule-based analysis. Heavy use of `asyncio` enables concurrent LLM calls and data processing. Source: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) Release history is captured in the GitHub Releases tab (`v1.0.0` "1.0 of tom-swe", `v1.0.1` "update typing", `v1.0.2` "Clean packages", `v1.0.3` "size optimization"). The installed version is exposed by `tom_swe.__init__.__version__`. Source: [tom_swe/__init__.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/__init__.py) Source: GitHub [Releases](https://github.com/OpenHands/ToM-SWE/releases) ## Known Gaps and Caveats - Issue [#30](https://github.com/OpenHands/ToM-SWE/issues/30) reports a dead link in `README.md` pointing to `tom_swe/prompts/registry.py`. Until it is repaired, callers should import prompts directly from `tom_swe.prompts.manager` (the path actually used by `tom_swe/tom_agent.py`). Source: GitHub issue [#30](https://github.com/OpenHands/ToM-SWE/issues/30) Source: [tom_swe/tom_agent.py](https://github.com/OpenHands/ToM-SWE/blob/main/tom_swe/tom_agent.py) - The REST/MCP endpoints listed above are specified by open issues but are not yet implemented in the codebase. - Issue [#29](https://github.com/OpenHands/ToM-SWE/issues/29) clarifies that the Stateful SWE benchmark data lives on Hugging Face (`cmu-lti/stateful`) rather than inside this repository. ## See Also - Project introduction and install instructions: [README.md](https://github.com/OpenHands/ToM-SWE/blob/main/README.md) - Pipeline architecture and key design patterns: [CLAUDE.md](https://github.com/OpenHands/ToM-SWE/blob/main/CLAUDE.md) - Release engineering and PyPI Trusted Publishing: [RELEASE.md](https://github.com/OpenHands/ToM-SWE/blob/main/RELEASE.md) - Related discussion: issues [#8](https://github.com/OpenHands/ToM-SWE/issues/8), [#10](https://github.com/OpenHands/ToM-SWE/issues/10), [#11](https://github.com/OpenHands/ToM-SWE/issues/11), [#14](https://github.com/OpenHands/ToM-SWE/issues/14), [#17](https://github.com/OpenHands/ToM-SWE/issues/17), [#29](https://github.com/OpenHands/ToM-SWE/issues/29), [#30](https://github.com/OpenHands/ToM-SWE/issues/30) --- <!-- evidence_pipeline_checked: true --> <!-- evidence_injected: true --> --- ## Pitfall Log Project: OpenHands/ToM-SWE Summary: Found 8 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification. ## 1. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.host_targets | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 2. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 3. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/OpenHands/ToM-SWE/issues/30 ## 4. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 5. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 6. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 7. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE ## 8. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:1010896499 | https://github.com/OpenHands/ToM-SWE <!-- canonical_name: OpenHands/ToM-SWE; human_manual_source: deepwiki_human_wiki -->