# https://github.com/cloudflare/agents Project Manual Generated at: 2026-06-14 02:43:40 UTC ## Table of Contents - [Core Agent Runtime, State & Routing](#page-1) - [AI Chat, Think Framework & Sub-agent Orchestration](#page-2) - [MCP, Code Mode, Tools & Extensibility](#page-3) - [Voice Pipeline, WebSockets & Frontend Clients](#page-4) ## Core Agent Runtime, State & Routing ### Related Pages Related topics: [AI Chat, Think Framework & Sub-agent Orchestration](#page-2), [Voice Pipeline, WebSockets & Frontend Clients](#page-4)
Related Source Files The following source files were used to generate this page: - [packages/agents/src/index.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/index.ts) - [packages/agents/src/client.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/client.ts) - [packages/agents/src/retries.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/retries.ts) - [packages/agents/src/schedule.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/schedule.ts) - [packages/agents/src/sub-routing.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/sub-routing.ts) - [packages/agents/src/types.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/types.ts) - [packages/agents/package.json](https://github.com/cloudflare/agents/blob/main/packages/agents/package.json) - [README.md](https://github.com/cloudflare/agents/blob/main/README.md) - [design/README.md](https://github.com/cloudflare/agents/blob/main/design/README.md) - [examples/playground/README.md](https://github.com/cloudflare/agents/blob/main/examples/playground/README.md) - [examples/mcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp/README.md) - [examples/dynamic-tools/README.md](https://github.com/cloudflare/agents/blob/main/examples/dynamic-tools/README.md)
# Core Agent Runtime, State & Routing ## Overview The Core Agent Runtime is the foundation of the Cloudflare Agents SDK, published as the `agents` package (v0.16.0) and implemented on top of Cloudflare Durable Objects ([packages/agents/package.json](https://github.com/cloudflare/agents/blob/main/packages/agents/package.json)). It packages a Durable Object subclass (`Agent`) with WebSocket/HTTP I/O, SQLite-backed state, scheduling, RPC, and MCP support, so an application can define a single class that owns its lifecycle, connections, persistence, and outbound calls ([README.md](https://github.com/cloudflare/agents/blob/main/README.md)). The roadmap confirms that HTTP/WebSockets, `this.state` / `setState` / `onStateUpdate`, RPC, scheduling, email, browser, and MCP are all built on this same runtime ([roadmap issue #2](https://github.com/cloudflare/agents/issues/2)). The routing layer decides *which* Durable Object instance handles a given request. `getAgentByName(Cls, name, props)` — supported on both the server and the client — resolves a stable name to an instance, optionally passing `props` that are serialized into the `x-partykit-props` header. Sub-routing lives in a dedicated module ([packages/agents/src/sub-routing.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/sub-routing.ts)) that exposes nested-URL routing for facets and child DOs ([design/sub-agent-routing.md](https://github.com/cloudflare/agents/blob/main/design/sub-agent-routing.md)). The client-side counterpart (`useAgent`, `useAgentChat`) lives in [packages/agents/src/client.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/client.ts). ## Lifecycle and Hooks The base `Agent` class exposes a small set of overridable lifecycle hooks that map directly onto Durable Object events: | Hook | Purpose | | --- | --- | | `onStart()` | Runs once when the DO is first constructed; use for one-time migrations or warm-up | | `onConnect(connection, ctx)` | Runs when a WebSocket client connects; the only place to attach a session/identity payload that must survive hibernation ([issue #321](https://github.com/cloudflare/agents/issues/321)) | | `onMessage(connection, message)` | Runs for every inbound WebSocket message | | `onClose(connection, code, reason)` | Runs when a client disconnects | | `onStateUpdate(state, source)` | Runs locally and across instances after `setState` | There is a known race: `onMessage()` can fire before `onConnect()` if a buffered message arrives before the connection's metadata settles ([issue #321](https://github.com/cloudflare/agents/issues/321)). The recommended fix is to **not** rely on connection attachments inside `onMessage`; instead, route any context-dependent logic through `onConnect` so identity is materialized before the first message can dispatch. `McpAgent`, the MCP-protocol subclass, layers its own `init()` for tool/resource registration ([examples/mcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp/README.md)). ## State, SQL, and Replicas `this.state` is the typed reactive state surface. Calling `setState(patch)` persists the patch to the DO's SQLite, broadcasts the update to every connected client and every peer replica, and fires `onStateUpdate` so other instances can react ([packages/agents/src/types.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/types.ts)). For richer persistence the runtime exposes `this.sql`, a template-tagged SQL helper that runs against the DO's embedded SQLite — the playground demonstrates this directly with a "SQL" demo ([examples/playground/README.md](https://github.com/cloudflare/agents/blob/main/examples/playground/README.md)). Search-style features are layered on top using FTS5 virtual tables; the experimental session-search package creates `cf_agents_search_entries` and `cf_agents_search_fts` ([experimental/session-search/README.md](https://github.com/cloudflare/agents/blob/main/experimental/session-search/README.md)). Be aware that some subclasses eagerly hydrate the full chat history in `onStart` — on a very long session this has been observed to throw `SQLITE_NOMEM`, permanently bricking the DO ([issue #1710](https://github.com/cloudflare/agents/issues/1710)). The optional `sanitizeMessageForPersistence()` hook lets subclasses redact content before it lands in storage, but there is currently no symmetric `sanitizeMessageForClient()` — community requests ask for that asymmetry to be closed ([issue #1744](https://github.com/cloudflare/agents/issues/1744)). ## Routing and Naming Strategies Routing in the Agents SDK is *name-based*: every `getAgentByName(Cls, name, props)` call resolves to the Durable Object whose name matches `name`, creating it on first use. The playground groups the patterns as **per-user**, **shared**, and **per-session** ([examples/playground/README.md](https://github.com/cloudflare/agents/blob/main/examples/playground/README.md)): | Strategy | Example name | Typical use | | --- | --- | --- | | Per-user | `user:${userId}` | One instance per authenticated user; full chat history isolation | | Shared | `global` or `lobby:${roomId}` | Multi-tenant rooms (chat rooms, lobbies) | | Per-session | `session:${sessionId}` | Throwaway or short-lived instances | ```mermaid flowchart LR Client[Client request] --> Hdr["x-partykit-props header"] Hdr --> Lookup[getAgentByName lookup] Lookup -->|hit| DO[Existing DO instance] Lookup -->|miss| New[Create DO with name] New --> Attach["onStart()"] Attach --> Ready[Ready to serve] DO --> Ready ``` Sub-routing goes further: a parent DO can spawn child DOs through a sub-routing API, exposing them at nested URLs and registering them with the parent's runtime ([design/sub-agent-routing.md](https://github.com/cloudflare/agents/blob/main/design/sub-agent-routing.md), [packages/agents/src/sub-routing.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/sub-routing.ts)). Child agents can be invoked as detached "background" runs, though first-class completion hooks for those detached runs are still an open request ([issue #1752](https://github.com/cloudflare/agents/issues/1752)). ### Known Routing Quirks - **`x-partykit-props` and non-ASCII.** `props` are serialized into a header; non-ASCII characters trigger a runtime warning about UTF-8 encoding ([issue #1751](https://github.com/cloudflare/agents/issues/1751)). Encode user-controlled strings before passing them as props, or keep them ASCII-only. - **Mount-time RPC.** `useAgent()` callers that fire `agent.call(...)` from a mount-time `useEffect` can hang because the response is dispatched before the RPC channel is fully wired ([issue #1738](https://github.com/cloudflare/agents/issues/1738)). Defer the first call until after a `connectionReady` signal or a user-gesture trigger. - **`waitUntil` on `McpAgent.destroy()`.** Calling `ctx.waitUntil(agent.destroy())` after the SSE client drops logs `waitUntil() tasks did not complete` ([issue #1625](https://github.com/cloudflare/agents/issues/1625)). Avoid wrapping teardown in `waitUntil` unless the client is known to still be attached. ## See Also - [packages/ai-chat](https://github.com/cloudflare/agents/blob/main/packages/ai-chat/README.md) — chat layer that builds on this runtime - [packages/think](https://github.com/cloudflare/agents/blob/main/packages/think/README.md) — opinionated chat agent base class - [design/think.md](https://github.com/cloudflare/agents/blob/main/design/think.md) — Think design notes - [design/sub-agent-routing.md](https://github.com/cloudflare/agents/blob/main/design/sub-agent-routing.md) — sub-agent routing deep dive - [guides/anthropic-patterns](https://github.com/cloudflare/agents/tree/main/guides/anthropic-patterns) — orchestrator / routing patterns --- ## AI Chat, Think Framework & Sub-agent Orchestration ### Related Pages Related topics: [Core Agent Runtime, State & Routing](#page-1), [MCP, Code Mode, Tools & Extensibility](#page-3), [Voice Pipeline, WebSockets & Frontend Clients](#page-4)
Related Source Files The following source files were used to generate this page: - [design/README.md](https://github.com/cloudflare/agents/blob/main/design/README.md) - [packages/ai-chat/README.md](https://github.com/cloudflare/agents/blob/main/packages/ai-chat/README.md) - [packages/agents/package.json](https://github.com/cloudflare/agents/blob/main/packages/agents/package.json) - [packages/codemode/src/resolve.ts](https://github.com/cloudflare/agents/blob/main/packages/codemode/src/resolve.ts) - [packages/codemode/src/json-schema-types.ts](https://github.com/cloudflare/agents/blob/main/packages/codemode/src/json-schema-types.ts) - [examples/ai-chat/README.md](https://github.com/cloudflare/agents/blob/main/examples/ai-chat/README.md) - [examples/agents-as-tools/README.md](https://github.com/cloudflare/agents/blob/main/examples/agents-as-tools/README.md) - [examples/multi-ai-chat/README.md](https://github.com/cloudflare/agents/blob/main/examples/multi-ai-chat/README.md) - [examples/chat-sdk-messenger/README.md](https://github.com/cloudflare/agents/blob/main/examples/chat-sdk-messenger/README.md) - [examples/dynamic-tools/README.md](https://github.com/cloudflare/agents/blob/main/examples/dynamic-tools/README.md) - [examples/structured-input/README.md](https://github.com/cloudflare/agents/blob/main/examples/structured-input/README.md) - [examples/assistant/README.md](https://github.com/cloudflare/agents/blob/main/examples/assistant/README.md) - [examples/mcp-elicitation/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp-elicitation/README.md)
# AI Chat, Think Framework & Sub-agent Orchestration ## Overview and Scope The Cloudflare Agents SDK ships three overlapping layers for AI-driven, stateful, multi-agent applications: a low-level `AIChatAgent` base class, a higher-level `Think` framework, and a sub-agent orchestration primitive (`agentTool` / `runAgentTool`). The design folder explicitly documents these as deliberate tradeoffs worth recording, with `think.md`, `agent-tools.md`, `sub-agent-routing.md`, and `rfc-sub-agents.md` covering the chat agent base class, sub-agent orchestration, routing via facets, and the RFC for child Durable Objects (DOs) respectively. Source: [design/README.md:9-15](). Together they form a stack where AI chat, durable execution, and multi-agent dispatch are composable rather than monolithic. ## AIChatAgent: Persistence, Resumable Streams, Tools `AIChatAgent` (`@cloudflare/ai-chat`) is the recommended entry point for single-conversation chat agents. The minimal server-side shape is to extend `AIChatAgent` and implement `onChatMessage`, returning `result.toUIMessageStreamResponse()` from `streamText`: Source: [packages/ai-chat/README.md:21-37](). The base class automatically persists messages to SQLite, ships them over WebSocket to all connected clients, and supports stream resumption on reconnect. Beyond the baseline, `AIChatAgent` exposes hooks for managing long-running sessions: `pruneMessages()` controls LLM-side context size, `maxPersistedMessages` caps storage growth, and `chatRecovery` enables Durable Object eviction recovery independently from client-side resumable streaming. Source: [examples/ai-chat/README.md:13-20](). On the client, `useAgentChat` accepts an `onToolCall` handler for client-side tools, an `addToolApprovalResponse` helper for `needsApproval` flows, and a `body` option for sending custom data with each request. Source: [examples/ai-chat/README.md:21-26](). A documented gap is that `sanitizeMessageForPersistence()` filters what is stored but no symmetric hook filters what is exposed to the client — for example, hiding a tool output from the frontend. The community has flagged this as issue #1744 ("Expose client-facing chat sanitization hooks separate from persistence sanitization"). For dynamic tool registration, `createToolsFromClientSchemas()` converts client-provided JSON Schemas into AI SDK tools, enabling SDK/platform patterns where the embedding app defines tools at runtime: Source: [examples/dynamic-tools/README.md:25-36](). ## Think Framework: Sessions, Streaming, and the Codemode Connector `Think` is the higher-level chat framework that layers durable sessions, fiber-based execution, and tool orchestration on top of the agent base. The `design/think.md` document covers the chat agent base class, sessions, streaming, tools, and the "execution ladder" that decides which tools a given turn may call: Source: [design/README.md:10](). Companion docs cover `think-durable-submissions.md` (async programmatic turns, recovery, idempotency), `agent-tools.md` (chat sub-agent orchestration and replay), and `workspace.md` (hybrid SQLite+R2 filesystem): Source: [design/README.md:10-14](). The `examples/chat-sdk-messenger` example shows the recommended embedding pattern: `ConversationAgent extends Think` owns AI history and model calls for one Chat SDK `thread.id`, delegating durable reply work to a managed fiber with an idempotency key: ```ts await this.startFiber( "chat-sdk-messenger:ai-reply", async (fiber) => { /* … */ }, { idempotencyKey: `ai-reply:${thread.id}:${message.id}` } ); ``` Source: [examples/chat-sdk-messenger/README.md:53-65](). The `@cloudflare/think@0.9.0` release rebuilt the Think `execute` tool on the codemode connector runtime, unifying how tools are resolved and adding built-in human-in-the-loop approvals. The codemode layer is framework-agnostic: `filterTools()` strips tools that require `needsApproval`, and `extractFns()` pulls `execute` functions out of tool records without any AI SDK or Zod dependency: Source: [packages/codemode/src/resolve.ts:18-49](). JSON Schema descriptors can be converted to TypeScript signatures via `generateTypesFromJsonSchema()`, supporting codemode use cases without AI SDK imports: Source: [packages/codemode/src/json-schema-types.ts:21-47](). A known operational failure mode is `Think.onStart` throwing `out of memory: SQLITE_NOMEM` on revival of long-lived sessions due to eager full-history hydration — community issue #1710. Because the throw happens in `onStart`, every subsequent wake (including alarm retries) re-throws, permanently bricking the DO. ## Sub-agent Orchestration Sub-agents are dispatched from a chat turn via two shipped primitives: `agentTool(SubAgent, …)` for ordinary LLM-selected helper calls, and `this.runAgentTool(SubAgent, …)` for explicit fan-out. Each invocation creates a real sub-agent DO with its own model, tools, messages, SQLite storage, and resumable chat stream. The parent forwards the child stream as `agent-tool-event` frames, and the React `useAgentToolEvents({ agent })` hook collects those frames for inline mini-chat UI: Source: [examples/agents-as-tools/README.md:13-21](). `clearAgentToolRuns()` deletes retained child facets when the parent chat is cleared. The multi-AI-chat example shows a stricter pattern where an `onBeforeSubAgent` gate enforces a registry before allowing the parent to dispatch into child `Chat` facets: Source: [examples/multi-ai-chat/README.md:62-74](). ```mermaid flowchart LR Browser -- ws --> Parent["Parent Chat DO"] Parent -- agentTool / runAgentTool --> Child["Sub-agent Facet DO"] Child -- agent-tool-event frames --> Parent Parent -- WS broadcast --> Browser Child -- own SQLite / stream --> Child ``` Community issue #1752 requests first-class detached ("background") sub-agent runs with a durable completion hook, so callers can fire-and-forget a sub-agent and receive a notification when its transcript completes — relevant because today's sub-agents are tightly coupled to the parent turn's lifecycle. Issue #368 asks for RPC APIs with streaming and cancellation for sub-agents, which would align sub-agent dispatch with the rest of the agent RPC surface. ## See Also - [README.md](https://github.com/cloudflare/agents/blob/main/README.md) — full example index, including `ai-chat/`, `multi-ai-chat/`, `agents-as-tools/`, `chat-sdk-messenger/`, `codemode/`, `codemode-mcp/`. - [design/think.md](https://github.com/cloudflare/agents/blob/main/design/think.md), [design/agent-tools.md](https://github.com/cloudflare/agents/blob/main/design/agent-tools.md), [design/sub-agent-routing.md](https://github.com/cloudflare/agents/blob/main/design/sub-agent-routing.md) — design rationale. - [examples/playground/README.md](https://github.com/cloudflare/agents/blob/main/examples/playground/README.md) — interactive demos for `setState`, `@callable`, `Fibers`, and `Supervisor` multi-agent patterns. --- ## MCP, Code Mode, Tools & Extensibility ### Related Pages Related topics: [AI Chat, Think Framework & Sub-agent Orchestration](#page-2)
Related Source Files The following source files were used to generate this page: - [examples/mcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp/README.md) - [examples/mcp-elicitation/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp-elicitation/README.md) - [examples/codemode-connectors/README.md](https://github.com/cloudflare/agents/blob/main/examples/codemode-connectors/README.md) - [examples/codemode-mcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/codemode-mcp/README.md) - [examples/codemode-mcp-openapi/README.md](https://github.com/cloudflare/agents/blob/main/examples/codemode-mcp-openapi/README.md) - [examples/webmcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/webmcp/README.md) - [examples/dynamic-tools/README.md](https://github.com/cloudflare/agents/blob/main/examples/dynamic-tools/README.md) - [examples/ai-chat/README.md](https://github.com/cloudflare/agents/blob/main/examples/ai-chat/README.md) - [packages/codemode/README.md](https://github.com/cloudflare/agents/blob/main/packages/codemode/README.md) - [packages/codemode/src/connectors/toolset.ts](https://github.com/cloudflare/agents/blob/main/packages/codemode/src/connectors/toolset.ts) - [packages/codemode/src/snippet.ts](https://github.com/cloudflare/agents/blob/main/packages/codemode/src/snippet.ts) - [packages/codemode/src/json-schema-types.ts](https://github.com/cloudflare/agents/blob/main/packages/codemode/src/json-schema-types.ts) - [README.md](https://github.com/cloudflare/agents/blob/main/README.md) - [design/README.md](https://github.com/cloudflare/agents/blob/main/design/README.md)
# MCP, Code Mode, Tools & Extensibility The Cloudflare Agents SDK exposes three overlapping extensibility surfaces that work together to give an LLM the ability to act on the outside world: the **Model Context Protocol (MCP)** for standardized server/client tool exchange, **Code Mode (`codemode`)** for sandboxed TypeScript execution that compresses N tools into one, and a **Tool/Connector extensibility layer** that wraps MCP servers, OpenAPI specs, AI-SDK tool sets, and client-defined tools behind a uniform runtime. The combination is what the SDK roadmap lists as "Shipped": MCP (server, client, OAuth, elicitation), Browser (rebuilt on codemode), and approval-gated tools, alongside experimental WebMCP bridging in the browser. Source: [README.md:1-40](). ## 1. MCP: Stateful Servers, Clients, and WebMCP `McpAgent` is the Agents SDK class for building MCP servers with persistent state. It extends `Agent`, so MCP tools and resources automatically get durable storage, scheduling, and `setState`/`onStateChanged` semantics inside a Durable Object. Source: [examples/mcp/README.md:1-30](). A minimal `McpAgent` looks like: ```typescript export class MyMCP extends McpAgent { server = new McpServer({ name: "Demo", version: "1.0.0" }); initialState: State = { counter: 1 }; async init() { this.server.registerTool("add", { /* schema */ }, async ({ a, b }) => ({ /* result */ })); this.server.resource("counter", "mcp://resource/counter", (uri) => ({ /* ... */ })); } } export default MyMCP.serve("/mcp", { binding: "MyMCP" }); ``` The default transport is **Streamable HTTP**, exposed at a named route via `McpAgent.serve(...)`. The `examples/mcp` demo exposes both a built-in tester UI and an MCP Inspector–compatible endpoint. Source: [examples/mcp/README.md:15-30](). Beyond the server, the SDK ships an MCP **client** (consuming remote MCP servers), OAuth, **elicitation** (server-prompted user input — see `examples/mcp-elicitation/`), and an experimental **WebMCP** adapter that bridges `McpAgent` tools to Chrome's `navigator.modelContext` API. Source: [examples/webmcp/README.md:1-40](). The adapter exposes a single `registerWebMcp({ url, prefix, getHeaders })` call, applies namespacing (e.g. `remote.greet`) to avoid collisions with page-local tools, listens for `tools/list_changed` notifications, and supports `dispose()` for cleanup. Source: [examples/webmcp/README.md:30-90](). Community caveat: `ctx.waitUntil(agent.destroy())` inside `McpAgent` session teardown can emit a "waitUntil() tasks did not complete" warning when the SSE client has already disconnected (issue #1625). This is a known nuisance in 0.13.3 and relates to the lifecycle ordering between transport close and DO shutdown. ## 2. Code Mode and the Connector Runtime **Code Mode** replaces the "N tool definitions, N round-trips" pattern with a single `code` (or `codemode`) tool: the model writes TypeScript that runs inside a sandboxed Worker, with connector SDKs exposed as globals. This collapses context usage and lets the model branch, loop, and reuse intermediate results in a single execution. Source: [packages/codemode/README.md:1-60]() and [examples/codemode-mcp/README.md:1-30](). ```mermaid flowchart LR User[User message] --> Agent[Agent.onChatMessage] Agent --> CodeTool["codemode tool
(single AI SDK tool)"] CodeTool --> Exec[Sandboxed Worker] Exec -->|typed calls| MCP[McpConnector] Exec -->|typed calls| OAPI[OpenApiConnector] Exec -->|typed calls| TS[ToolSetConnector] MCP -->|list/describe/run| Sandbox OAPI -->|host-derived ops| Sandbox TS -->|AI SDK tools| Sandbox Sandbox -->|stream| Agent Agent -->|UI stream| User ``` The connector layer is the extensibility spine. `McpConnector` wraps an MCP server and derives one typed sandbox method per upstream tool; `OpenApiConnector` reads the spec **once, host-side** and derives one typed tool per operation; `ToolSetConnector` adapts an existing AI SDK `ToolSet`. Source: [packages/codemode/README.md:60-130]() and [examples/codemode-connectors/README.md:1-60](). `ToolSetConnector` enforces a key invariant: **only tools that have an `execute` function are exposed to the sandbox** — client-side/provider-executed tools are filtered out, because advertising a method the sandbox cannot call would send the model down a dead end. Source: [packages/codemode/src/connectors/toolset.ts:30-60](). Approval-gated writes are first-class: a connector can mark a tool (e.g. `github.create_issue`) as `requiresApproval`, which pauses the run; the agent exposes `pendingApprovals`, `approveExecution(id)`, and `rejectExecution(id)` as `@callable()` methods that the UI wires to buttons. Approving re-runs the stored code, replaying everything up to the approved action. Source: [examples/codemode-connectors/README.md:30-70](). ### Snippets and durable capabilities Working executions can be promoted into **snippets** — durable, named scripts addressable by `codemode.run("name", input)`. Each snippet records its name, description, source, save timestamp, optional input schema, and the connector set it ran with; subsequent `run`s verify connectors are still configured. Source: [packages/codemode/src/snippet.ts:1-50]()`. OpenAPI integrations get a turnkey server via `openApiMcpServer({ spec, executor, request })`, which exposes a `search` tool (model queries the spec as a JS object) and an `execute` tool (host-side `request()` does the authenticated call, so secrets never enter the sandbox). Source: [examples/codemode-mcp-openapi/README.md:1-40](). ## 3. Tooling Patterns and JSON Schema Interop The codemode package ships a JSON-Schema–only type generator for cases where there is no AI-SDK or Zod dependency (e.g. raw OpenAPI or MCP descriptors). `generateTypesFromJsonSchema(tools)` emits input/output TypeScript types from a `JsonSchemaToolDescriptors` map, deriving JSDoc `@param` annotations from each schema's descriptions. Source: [packages/codemode/src/json-schema-types.ts:1-50](). Dynamic, **client-defined tools** round-trip the other direction: a generic server accepts tool schemas from the embedding client over WebSocket and rebuilds AI SDK tools via `createToolsFromClientSchemas(options.clientTools)`. This is the platform/SDK pattern — tools are unknown at deploy time. Source: [examples/dynamic-tools/README.md:1-50](). The agent tools layer is the chat sub-agent orchestration primitive covered in design docs (`design/agent-tools.md`, `design/sub-agent-routing.md`); it is rebuilt on the codemode connector runtime in `agents@0.16.0`, and `Think`'s execute tool likewise moved onto codemode in `@cloudflare/think@0.9.0` for unified approvals and replay. Source: [design/README.md:1-20]() and release notes for `agents@0.16.0`. ## 4. Known Failure Modes and Roadmap Signals | Surface | Symptom / request | Source | | --- | --- | --- | | MCP server teardown | `waitUntil() tasks did not complete` warning when SSE client drops first | issue #1625 | | Sub-agent runs | Request for first-class "background" runs with a durable completion hook | issue #1752 | | Chat sanitization | Need a client-facing hook separate from `sanitizeMessageForPersistence` | issue #1744 | | Sub-agent RPC | RPC API for streaming/cancellation on `runAgentTool` | issue #321, #368 | | `useAgent` RPC | `agent.call(...)` from mount-time `useEffect` can hang forever | issue #1738 | These map directly onto the SDK's stated "Shipped" list and the still-open RFC items (e.g. thread/branch sessions in #1075, AI SDK 5 alignment in #343). ## See Also - Stateful MCP Server — [examples/mcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/mcp/README.md) - Code Mode package — [packages/codemode/README.md](https://github.com/cloudflare/agents/blob/main/packages/codemode/README.md) - WebMCP Adapter — [examples/webmcp/README.md](https://github.com/cloudflare/agents/blob/main/examples/webmcp/README.md) - Design decisions — [design/README.md](https://github.com/cloudflare/agents/blob/main/design/README.md) - Repository overview — [README.md](https://github.com/cloudflare/agents/blob/main/README.md) --- ## Voice Pipeline, WebSockets & Frontend Clients ### Related Pages Related topics: [Core Agent Runtime, State & Routing](#page-1), [AI Chat, Think Framework & Sub-agent Orchestration](#page-2)
Related Source Files The following source files were used to generate this page: - [packages/agents/src/react.tsx](https://github.com/cloudflare/agents/blob/main/packages/agents/src/react.tsx) - [packages/agents/src/index.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/index.ts) - [packages/voice/src/voice.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice.ts) - [packages/voice/src/voice-client.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-client.ts) - [packages/voice/src/voice-input.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-input.ts) - [packages/voice/src/voice-react.tsx](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-react.tsx) - [packages/voice/README.md](https://github.com/cloudflare/agents/blob/main/packages/voice/README.md) - [README.md](https://github.com/cloudflare/agents/blob/main/README.md)
# Voice Pipeline, WebSockets & Frontend Clients The Cloudflare Agents SDK provides an end-to-end transport layer for real-time agent communication, composed of three cooperating pieces: a WebSocket-aware `Agent` Durable Object on the server, an `@cloudflare/voice` pipeline that wraps that agent with STT, TTS, and interruption handling, and a set of React/vanilla clients that maintain a live socket, multiplex RPC and chat over it, and synthesize playback in the browser. This page documents how these pieces fit together and where they tend to break. ## Architecture Overview The `Agent` base class extends a Cloudflare Durable Object with WebSocket lifecycle hooks (`onConnect`, `onMessage`, `onClose`, `onError`) and a state-sync channel (`setState` / `onStateUpdate`). Source: [packages/agents/src/index.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/index.ts). On top of that, `@cloudflare/voice` exports `withVoice(Agent)`, which produces a mixin that owns the transcriber, the TTS provider, the call lifecycle, and a protocol-aware `onMessage` that intercepts voice frames before they reach user code. Source: [packages/voice/src/voice.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice.ts). ```mermaid flowchart LR Browser[VoiceClient / useAgent] -- WS frames --> DO[VoiceAgent DO] DO --> STT[STT provider] STT --> Hooks[onTurn / afterTranscribe] Hooks --> LLM[LLM call] LLM --> Hooks2[beforeSynthesize] Hooks2 --> TTS[TTS provider] TTS -- audio chunks --> Browser Browser --> AudioOut[Web Audio playback] ``` The protocol is the same `CF_AGENT` wire format used by AI chat — voice just adds framed audio and `transcript_interim` messages. The frontend clients speak this protocol automatically; consumers rarely need to read the raw frames. ## Server-Side Voice Pipeline `withVoice` injects the call lifecycle and pipeline hooks documented in [packages/voice/README.md](https://github.com/cloudflare/agents/blob/main/packages/voice/README.md). Subclasses are required to implement `onTurn(transcript, context)` and assign a `transcriber` (e.g. `DeepgramSTT`, `TelnyxSTT`, the built-in `WorkersAITranscriber`) and a `tts` (e.g. `WorkersAITTS`, `ElevenLabsTTS`). The pipeline stages `afterTranscribe`, `beforeSynthesize`, and `afterSynthesize` are override points that may return `null` to short-circuit. A call begins when a client opens a WebSocket; `beforeCallStart` may return `false` to reject. Once accepted, audio flows client → DO → STT provider. Partial transcripts are forwarded as `transcript_interim` events so the UI can render "I'm listening..." indicators. Final transcripts trigger `afterTranscribe`, then `onTurn`, then `beforeSynthesize` and `afterSynthesize` before audio is streamed back. Interruption is detected either from client-side audio-level analysis or from the model's speech-start signal and surfaces as the `onInterrupt(connection)` hook. Source: [packages/voice/src/voice.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice.ts). The voice input utilities ([packages/voice/src/voice-input.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-input.ts)) handle resampling, VAD, and chunk framing so consumers do not have to reimplement them. ## WebSocket Transport & RPC All real-time features share one WebSocket. The `Agent` class multiplexes four flows over that socket: state patches, RPC calls (any `@callable()` method), chat streaming, and voice frames. State is sent on `setState` and replayed to the connection on `onConnect`; if the DO hibernates and wakes, the new instance re-attaches and state resyncs. Source: [packages/agents/src/index.ts](https://github.com/cloudflare/agents/blob/main/packages/agents/src/index.ts). The React client wraps this in `useAgent({ agent, name, host })`, returning a typed proxy with `.state`, `.setState`, and `.call(...)`. Source: [packages/agents/src/react.tsx](https://github.com/cloudflare/agents/blob/main/packages/agents/src/react.tsx). On the voice side, `useVoiceAgent({ agent, name, host })` extends that with `startCall()`, `endCall()`, `interimTranscript`, and an `outputDeviceId` / `setOutputDevice()` API for routing assistant playback to a selected sink (added in `@cloudflare/voice@0.3.0`). Source: [packages/voice/src/voice-react.tsx](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-react.tsx). For non-React callers, `VoiceClient` and `AgentClient` are vanilla equivalents. Source: [packages/voice/src/voice-client.ts](https://github.com/cloudflare/agents/blob/main/packages/voice/src/voice-client.ts). ## Known Failure Modes & Usage Pitfalls The community has surfaced several recurring issues worth documenting up front: - **Audio click at chunk seams.** `VoiceClient` previously started each chunk at the current playback cursor and awaited its `ended` event before scheduling the next, leaving ~5 ms of silence per seam. Issue [#1736](https://github.com/cloudflare/agents/issues/1736) includes a fix that schedules the next chunk immediately. - **RPC hang during initial connect.** `agent.call(...)` issued from a mount-time `useEffect` can hang forever because the request is sent before the socket's RPC channel is ready. Later calls succeed. Issue [#1738](https://github.com/cloudflare/agents/issues/1738) describes the symptom; the workaround is to defer the first call until after the connection is established. - **onMessage before onConnect race.** `onMessage()` may fire before `onConnect()` on a hibernated DO wake, so identity attachments must be re-fetched defensively rather than cached on the connection. Issue [#321](https://github.com/cloudflare/agents/issues/321). - **Non-ASCII in `x-partykit-props`.** `getAgentByName(Agent, id, props)` logs a warning when `props` contains non-ASCII characters because of how the SDK encodes them as UTF-8 in the `x-partykit-props` header. Issue [#1751](https://github.com/cloudflare/agents/issues/1751). - **Eager hydration on large sessions.** `Think.onStart` historically hydrated full history on revive; for large sessions this throws `SQLITE_NOMEM` and bricks the DO. Issue [#1710](https://github.com/cloudflare/agents/issues/1710). ## See Also - [README.md](https://github.com/cloudflare/agents/blob/main/README.md) — package catalog and feature matrix - [packages/voice/README.md](https://github.com/cloudflare/agents/blob/main/packages/voice/README.md) — full voice lifecycle reference - [examples/voice-agent/](https://github.com/cloudflare/agents/blob/main/examples/voice-agent/) — end-to-end voice example - [examples/elevenlabs-starter/](https://github.com/cloudflare/agents/blob/main/examples/elevenlabs-starter/) — ElevenLabs integration with voice chat, soundscape, character, and music agents - [voice-providers/deepgram/](https://github.com/cloudflare/agents/blob/main/voice-providers/deepgram/) and [voice-providers/telnyx/](https://github.com/cloudflare/agents/blob/main/voice-providers/telnyx/) — STT/TTS provider adapters --- --- ## Pitfall Log Project: cloudflare/agents Summary: Found 30 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Maintenance risk - Maintenance risk requires verification. ## 1. Maintenance risk - Maintenance risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1736 ## 2. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this installation risk before relying on the project: agents@0.15.0 - User impact: Upgrade or migration may change expected behavior: agents@0.15.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/agents%400.15.0 ## 3. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this installation risk before relying on the project: create-think@0.0.3 - User impact: Upgrade or migration may change expected behavior: create-think@0.0.3 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/create-think%400.0.3 ## 4. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this installation risk before relying on the project: create-think@0.0.4 - User impact: Upgrade or migration may change expected behavior: create-think@0.0.4 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/create-think%400.0.4 ## 5. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: @cloudflare/codemode@0.4.0 - User impact: Upgrade or migration may change expected behavior: @cloudflare/codemode@0.4.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/%40cloudflare/codemode%400.4.0 ## 6. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: @cloudflare/think@0.9.0 - User impact: Upgrade or migration may change expected behavior: @cloudflare/think@0.9.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/%40cloudflare/think%400.9.0 ## 7. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: McpAgent: ctx.waitUntil(agent.destroy()) on session teardown logs "waitUntil() tasks did not complete" when the SSE client has already disconnected (0.13.3) - User impact: Developers may misconfigure credentials, environment, or host setup: McpAgent: ctx.waitUntil(agent.destroy()) on session teardown logs "waitUntil() tasks did not complete" when the SSE client has already disconnected (0.13.3) - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1625 ## 8. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this configuration risk before relying on the project: [retracted] - User impact: Developers may misconfigure credentials, environment, or host setup: [retracted] - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1749 ## 9. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1738 ## 10. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | github_repo:924394244 | https://github.com/cloudflare/agents ## 11. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this runtime risk before relying on the project: @cloudflare/shell@0.4.0 - User impact: Upgrade or migration may change expected behavior: @cloudflare/shell@0.4.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/%40cloudflare/shell%400.4.0 ## 12. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this runtime risk before relying on the project: A header value for "x-partykit-props" contains non-ASCII characters - User impact: Developers may hit a documented source-backed failure mode: A header value for "x-partykit-props" contains non-ASCII characters - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1751 ## 13. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this runtime risk before relying on the project: useAgent: RPC calls issued during initial connect can hang forever (response silently dropped) - User impact: Developers may hit a documented source-backed failure mode: useAgent: RPC calls issued during initial connect can hang forever (response silently dropped) - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1738 ## 14. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1751 ## 15. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1710 ## 16. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: packet_text.keyword_scan | github_repo:924394244 | https://github.com/cloudflare/agents ## 17. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Developers should check this migration risk before relying on the project: agents@0.16.0 - User impact: Upgrade or migration may change expected behavior: agents@0.16.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/agents%400.16.0 ## 18. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:924394244 | https://github.com/cloudflare/agents ## 19. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | github_repo:924394244 | https://github.com/cloudflare/agents ## 20. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | github_repo:924394244 | https://github.com/cloudflare/agents ## 21. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1752 ## 22. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1625 ## 23. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/cloudflare/agents/issues/1749 ## 24. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: Expose client-facing chat sanitization hooks separate from persistence sanitization - User impact: Developers may hit a documented source-backed failure mode: Expose client-facing chat sanitization hooks separate from persistence sanitization - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1744 ## 25. Capability evidence risk - Capability evidence risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this capability risk before relying on the project: [voice] audible click at every audio chunk seam (w/ working fix) - User impact: Developers may hit a documented source-backed failure mode: [voice] audible click at every audio chunk seam (w/ working fix) - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1736 ## 26. Runtime risk - Runtime risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this performance risk before relying on the project: Feature request: first-class detached ("background") sub-agent runs with a durable completion hook - User impact: Developers may hit a documented source-backed failure mode: Feature request: first-class detached ("background") sub-agent runs with a durable completion hook - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1752 ## 27. Runtime risk - Runtime risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this performance risk before relying on the project: Think.onStart bricks the DO with SQLITE_NOMEM on large sessions (eager full-history hydration) - User impact: Developers may hit a documented source-backed failure mode: Think.onStart bricks the DO with SQLITE_NOMEM on large sessions (eager full-history hydration) - Evidence: failure_mode_cluster:github_issue | https://github.com/cloudflare/agents/issues/1710 ## 28. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:924394244 | https://github.com/cloudflare/agents ## 29. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | github_repo:924394244 | https://github.com/cloudflare/agents ## 30. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: Developers should check this maintenance risk before relying on the project: @cloudflare/voice@0.3.0 - User impact: Upgrade or migration may change expected behavior: @cloudflare/voice@0.3.0 - Evidence: failure_mode_cluster:github_release | https://github.com/cloudflare/agents/releases/tag/%40cloudflare/voice%400.3.0