# https://github.com/conductor-oss/conductor Project Manual Generated at: 2026-06-19 03:07:26 UTC ## Table of Contents - [Conductor Overview & System Architecture](#page-1) - [Workflow Engine, Tasks & Execution Model](#page-2) - [AI, LLM, MCP & Vector Database Integration](#page-3) - [Deployment, Configuration & Troubleshooting](#page-4) ## Conductor Overview & System Architecture ### Related Pages Related topics: [Workflow Engine, Tasks & Execution Model](#page-2), [Deployment, Configuration & Troubleshooting](#page-4)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) - [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md) - [kafka/README.md](https://github.com/conductor-oss/conductor/blob/main/kafka/README.md) - [ai/examples/README.md](https://github.com/conductor-oss/conductor/blob/main/ai/examples/README.md) - [common/src/main/java/com/netflix/conductor/common/metadata/workflow/CacheConfig.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/workflow/CacheConfig.java) - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/ExecutionMetadata.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/ExecutionMetadata.java) - [common/src/main/java/com/netflix/conductor/common/run/WorkflowSummaryExtended.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/run/WorkflowSummaryExtended.java) - [common/src/main/java/org/conductoross/conductor/model/SignalResponse.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/SignalResponse.java) - [common/src/main/java/org/conductoross/conductor/model/WorkflowSignalReturnStrategy.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/WorkflowSignalReturnStrategy.java) - [common/src/main/java/org/conductoross/conductor/model/WorkflowRun.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/WorkflowRun.java) - [common/src/main/java/org/conductoross/conductor/model/TaskRun.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/TaskRun.java) - [common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresBaseDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresBaseDAO.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresExecutionDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresExecutionDAO.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresIndexDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresIndexDAO.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresQueueDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresQueueDAO.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresProperties.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresProperties.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresConfiguration.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresConfiguration.java)
# Conductor Overview & System Architecture ## 1. Purpose and Scope Conductor is an open-source workflow orchestration engine originally created at Netflix and now maintained as the `conductor-oss` project under the Apache 2.0 license. The top-level [README.md:1-30](https://github.com/conductor-oss/conductor/blob/main/README.md) positions Conductor as a platform for authoring, executing, and observing durable, stateful workflows that coordinate heterogeneous tasks (HTTP calls, microservices, AI agents, scripts, message-queue work, and more). The repository ships: - A **server** module exposing REST APIs and the orchestration runtime. - A **common** module holding shared data models used by both server and client SDKs. - Pluggable **persistence** modules — one of which is the Postgres-backed implementation in `postgres-persistence/`. - Optional **community-contributed** integrations such as the Kafka system tasks described in [kafka/README.md:1-15](https://github.com/conductor-oss/conductor/blob/main/kafka/README.md). - A growing set of **AI workflow examples** (LLM chat, code agents, research agents) under [ai/examples/README.md:1-25](https://github.com/conductor-oss/conductor/blob/main/ai/examples/README.md) that illustrate how to register and execute LLM-driven workflows through the same REST surface. A separate project, [conductor-oss/rust-sdk](https://github.com/conductor-oss/rust-sdk), provides an incubating Rust client SDK, reflecting the project's multi-language direction. ## 2. High-Level Architecture Conductor follows a layered, interface-driven design. The orchestration engine is decoupled from any specific datastore, and persistence is selected at boot time through Spring conditional configuration. ```mermaid flowchart TB Client["Client / SDK / UI"] -->|REST + JSON| Server["Conductor Server
(Spring Boot)"] Server --> Core["Core Orchestration
(WorkflowExecutor, DeciderService,
SystemTask workers)"] Core --> DAO["DAO Interfaces
(MetadataDAO, ExecutionDAO,
QueueDAO, IndexDAO)"] DAO --> Pg["Postgres DAO
(PostgresMetadataDAO,
PostgresExecutionDAO,
PostgresQueueDAO,
PostgresIndexDAO)"] Pg --> Db[("PostgreSQL")] Core --> Async["Async Queues
(PostgresQueueDAO listener)"] Async --> Pg ``` The `common` module defines the public-facing data models — `WorkflowDef`, `TaskDef`, `Workflow`, `Task`, `TaskResult`, and friends — and ships JSON Schemas for client-side validation. Source: [schemas/README.md:1-30](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md). The recursive `WorkflowTask` schema enables unlimited nesting of decision, fork-join, do-while, and sub-workflow patterns. The same models are surfaced through Protobuf annotations (e.g. `@ProtoMessage` / `@ProtoField`) on classes such as [CacheConfig.java:13-22](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/workflow/CacheConfig.java) and [ExecutionMetadata.java:19-26](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/ExecutionMetadata.java), which is why the runtime can support both JSON REST and binary gRPC transports from a single source of truth. ### Pluggable persistence The `conductor.db.type` Spring property gates which DAO implementation is wired in. For Postgres, see the conditional bean in [PostgresConfiguration.java:42-66](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresConfiguration.java) (`@ConditionalOnProperty(name = "conductor.db.type", havingValue = "postgres")`), and the toggleable properties in [PostgresProperties.java:18-44](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/config/PostgresProperties.java) (task-def cache refresh, deadlock retry max, queue-notify toggles, data-migration flag). The README enumerates the supported backend combinations in [README.md:1-50](https://github.com/conductor-oss/conductor/blob/main/README.md): Redis with ES7/ES8/OpenSearch, and Postgres-only or Postgres-plus-ES variants. ## 3. Persistence Layer (Postgres Implementation) The Postgres persistence module implements the four core DAO contracts through one shared base class and four concrete DAOs. - **[PostgresBaseDAO.java:36-70](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresBaseDAO.java)** — supplies the shared `DataSource`, `ObjectMapper`, and a `RetryTemplate` used for deadlock/connection retries; it also filters out the DAO's own stack frames from logged exceptions. - **[PostgresMetadataDAO.java:1-40](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java)** — stores `WorkflowDef`, `TaskDef`, and `EventHandler` definitions, with an in-memory cache refreshed every `taskDefCacheRefreshInterval` seconds (default 60s). - **[PostgresExecutionDAO.java:1-40](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresExecutionDAO.java)** — persists workflow and task state, plus the rate-limiting and concurrent-execution limit tables; it pushes events through `Monitors` for metrics. - **[PostgresQueueDAO.java:1-40](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresQueueDAO.java)** — implements task queues; uses a `PostgresQueueListener` for in-process notifications when `experimentalQueueNotify=true` and unacks stalled messages after 60s. - **[PostgresIndexDAO.java:1-40](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresIndexDAO.java)** — provides searchable indices of workflows, tasks, and `TaskExecLog` entries without requiring an external search engine. > **Community note:** the analogous Elasticsearch 7 path has a long-standing complaint (issue #226) that `createIndexesTemplates` produced an unbounded number of `conductor_task_log_YYYYMMDD` indices, eventually exhausting shards. The Postgres `IndexDAO` sidesteps that by storing the log in relational tables. ## 4. Core Data Models & Execution Surface The `common` module defines the contracts that flow over the wire. Beyond the legacy `Workflow` / `Task` shapes, several newer models are worth knowing: - **[WorkflowSummaryExtended.java:18-40](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/run/WorkflowSummaryExtended.java)** — extends `WorkflowSummary` so that `input` and `output` are returned as `Map` for JSON consumers, while staying Protobuf-compatible via dedicated field ids. - **[WorkflowRun.java:1-25](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/WorkflowRun.java)** and **[TaskRun.java:1-30](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/TaskRun.java)** — DTOs used by the signal-handling endpoints to return *what is currently blocking* a workflow or task. Both extend the abstract **[SignalResponse.java:15-30](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/SignalResponse.java)**, which carries `workflowId`, `correlationId`, `requestId`, and the chosen return strategy. - **[WorkflowSignalReturnStrategy.java:8-30](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/org/conductoross/conductor/model/WorkflowSignalReturnStrategy.java)** — enum selecting between `TARGET_WORKFLOW`, `BLOCKING_WORKFLOW`, `BLOCKING_TASK`, and `BLOCKING_TASK_INPUT`. This is the key control for how deep into nested sub-workflows a signal response should be materialised. The accompanying comment notes this was a non-trivial design point once the notification service was introduced. - **[WorkflowTestRequest.java:1-40](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java)** — request body for the workflow *test* endpoint; supports per-task mocks (`taskRefToMockOutput`) and per-sub-workflow mocks (`subWorkflowTestRequest`), plus simulated `executionTime` and `queueWaitTime` for timeout tests. - **[CacheConfig.java:13-40](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/workflow/CacheConfig.java)** — per-workflow response-cache configuration (`key` + `ttlInSecond`). ## 5. Build, Runtime, and Operational Gotchas Per [README.md:1-60](https://github.com/conductor-oss/conductor/blob/main/README.md), building from source requires **JDK 21+**, Docker Desktop, Node.js 18+, and pnpm (for the UI). Community issue #711 highlights that the bundled `docker/server/Dockerfile` historically referenced Java 17 in two places, which conflicts with the `JavaLanguageVersion.of(21)` toolchain declared in `build.gradle`. Operators upgrading from older Conductor 3.21.x should also be aware of issue #712, in which parent workflows can hang after a sub-workflow completes and require a manual `POST /api/workflow/decide/{id}` to unblock. Elasticsearch users should set `conductor.indexing.type=elasticsearch` explicitly (issue #928) so that the ES7 `IndexDAO` bean is registered; without it Spring's auto-configuration can fail to start the server. For ES8 (issue #49), use the dedicated [config-redis-es8.properties](https://github.com/conductor-oss/conductor/blob/main/docker/server/config/config-redis-es8.properties) profile. The latest release line (v3.31.0-rc.4) ships the new `ui-next` front-end, an updated FAQ entry for duplicate task scheduling on attempt 0, and test-harness fixes — see the [release notes](https://github.com/conductor-oss/conductor/releases) for the full change set. ## See Also - [Schemas Reference](https://conductor-oss.org) — JSON Schemas for `WorkflowDef` and `TaskDef` - [Postgres Persistence Configuration](https://conductor-oss.org) — tuning guide for `PostgresProperties` - [Signal API & `WorkflowSignalReturnStrategy`](https://conductor-oss.org) — deep-dive on the signal response models - [Kafka Community Tasks](https://github.com/conductor-oss/conductor/blob/main/kafka/README.md) — system tasks contributed under the `conductor-task` artifact - [AI Workflow Examples](https://github.com/conductor-oss/conductor/blob/main/ai/examples/README.md) — end-to-end LLM-agent workflow samples --- ## Workflow Engine, Tasks & Execution Model ### Related Pages Related topics: [Conductor Overview & System Architecture](#page-1), [AI, LLM, MCP & Vector Database Integration](#page-3)
Related Source Files The following source files were used to generate this page: - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java) - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) - [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md) - [common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java) - [common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java) - [kafka/README.md](https://github.com/conductor-oss/conductor/blob/main/kafka/README.md) - [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) - [ai/examples/README.md](https://github.com/conductor-oss/conductor/blob/main/ai/examples/README.md) - [annotations-processor/README.md](https://github.com/conductor-oss/conductor/blob/main/annotations-processor/README.md) - [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md) - [scheduler/examples/README.md](https://github.com/conductor-oss/conductor/blob/main/scheduler/examples/README.md)
# Workflow Engine, Tasks & Execution Model ## Purpose and Scope Conductor's workflow engine orchestrates long-running, distributed business processes by interpreting declarative workflow definitions and driving tasks through a well-defined lifecycle. A workflow definition (the *blueprint*) is registered once and used to spin off many runtime workflow *instances*. Each instance schedules and tracks the tasks defined inside it, dispatching them to external workers (HTTP, gRPC, polling), system tasks (Kafka, JQ, HTTP, LLM), or built-in control-flow tasks (DECISION, SWITCH, FORK_JOIN, DO_WHILE, SUB_WORKFLOW). Source: [schemas/README.md](schemas/README.md) describes the four core schemas — `WorkflowDef`, `TaskDef`, `Workflow`, and `Task` — that model this two-layer split between template and runtime. Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java](common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java) enumerates every built-in task kind Conductor knows how to interpret, including simple, control-flow, integration, and LLM/MCP tasks. ## Core Data Model ### Definitions vs. Instances | Schema / Class | Role | Notes | |---|---|---| | `WorkflowDef` | Blueprint of a workflow | Recursive `WorkflowTask` allows unlimited nesting of DECISION/SWITCH/FORK_JOIN/DO_WHILE/SUB_WORKFLOW Source: [schemas/README.md](schemas/README.md) | | `TaskDef` | Reusable task template | Carries retry count/delay, timeout policy (`RETRY`/`TIME_OUT_WF`/`ALERT_ONLY`), and rate-limit settings Source: [schemas/README.md](schemas/README.md) | | `Workflow` | Live runtime instance | Carries status, task list, history (recursive for versioning), priority 0–99, parent/child links Source: [schemas/README.md](schemas/README.md) | | `Task` | Live runtime task | Carries status, input/output, workerId, retry counters, iteration, subWorkflowId Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java](common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) | Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java](common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) shows `Task` uses `@ProtoField` annotations for compact binary serialization while remaining compatible with JSON via the proto-aware Jackson module. ### Task Statuses Source: [schemas/README.md](schemas/README.md) defines the runtime state machine: - **In-flight:** `SCHEDULED`, `IN_PROGRESS` - **Success:** `COMPLETED`, `COMPLETED_WITH_ERRORS`, `SKIPPED` - **Failure:** `FAILED`, `FAILED_WITH_TERMINAL_ERROR`, `TIMED_OUT`, `CANCELED` Workflow-level statuses are `RUNNING`, `COMPLETED`, `FAILED`, `TIMED_OUT`, `TERMINATED`, `PAUSED`. ## Task Types and Execution Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java](common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java) enumerates every supported task kind. The enum is mirrored as string constants (e.g. `TASK_TYPE_HTTP = "HTTP"`) so call sites never hardcode literals. Categories present in the enum include: | Category | Examples | |---|---| | Simple / Worker | `SIMPLE`, `HTTP`, `INLINE`, `LAMBDA` | | Control-flow | `DECISION`, `SWITCH`, `JOIN`, `EXCLUSIVE_JOIN`, `FORK_JOIN`, `FORK_JOIN_DYNAMIC`, `DO_WHILE`, `EVENT`, `WAIT`, `HUMAN` | | Sub-workflows | `SUB_WORKFLOW`, `START_WORKFLOW` | | AI / LLM | `LLM_TEXT_COMPLETE`, `LLM_CHAT_COMPLETE`, `LLM_INDEX_TEXT`, `LLM_SEARCH_INDEX`, `LLM_GENERATE_EMBEDDINGS`, `LLM_STORE_EMBEDDINGS`, `LLM_GET_EMBEDDINGS` | | MCP / agentic | `LIST_MCP_TOOLS`, `CALL_MCP_TOOL`, `PULL_WORKFLOW_MESSAGES` | Source: [kafka/README.md](kafka/README.md) shows how community-contributed system tasks like `KAFKA_PUBLISH` and `JSON_JQ_TRANSFORM_TASK` are modeled — a task declares its `type` and an `inputParameters` map. JQ tasks, for example, must include a `queryExpression` plus any user-named variables. Source: [ai/examples/README.md](ai/examples/README.md) illustrates end-to-end execution: a workflow such as `chat_workflow` is registered via `POST /api/metadata/workflow`, started via `POST /api/workflow/{name}`, and inspected via `GET /api/workflow/{workflowId}`. Inputs and outputs are passed as JSON objects, and outputs from earlier tasks are referenced by `${taskRefName.output.field}` placeholders. ## Execution Flow ```mermaid flowchart LR A[WorkflowDef registered] --> B[POST /api/workflow/name] B --> C[Workflow instance created] C --> D[Engine maps WorkflowTasks -> Tasks] D --> E{Task type} E -->|Simple/HTTP| F[Worker polls & completes] E -->|System task| G[In-process executor
HTTP, JQ, Kafka, LLM] E -->|Control-flow| H[Branch / fork / loop / sub-workflow] F & G & H --> I[Task status updated] I --> J{Workflow complete?} J -->|No| D J -->|Yes| K[Workflow terminal status] ``` Source: [schemas/README.md](schemas/README.md) confirms that nested control-flow tasks — `decisionCases`, `defaultCase`, `forkTasks`, `loopOver`, `loopCondition`, `subWorkflowParam` — are encoded directly in the recursive `WorkflowTask` definition. Source: [common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java](common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java) shows how events plug into execution: an `EventHandler` carries an `actions` list and a nested `TaskDetails` message that references the producing `workflowId`, `taskRefName`, `taskId`, and `output`, enabling event-driven task completion without polling. ## Sub-Workflow and Async Patterns Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java](common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) reserves fields for sub-workflow coordination: `subWorkflowId` and `workflowInstanceId` link a `SUB_WORKFLOW` task to the child workflow it spawned, while `workflowPriority` lets high-priority runs preempt lower ones. A common community-reported failure mode (referenced in issue #712 *"The workflow hangs after running Sub workflow"*) is that the parent workflow does not automatically resume after the child completes until an external `POST /api/workflow/decide/{workflowId}` is issued. This is consistent with the explicit linking fields in `Task` and emphasizes that sub-workflow completion must be observed through the normal async-decision loop. ## Serialization and Wire Format Source: [common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java](common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java) explains how `Any` protobufs (used for `inputMessage` and `outputMessage` on `Task`, and `outputMessage` on `EventHandler.TaskDetails`) are exposed over REST without requiring the server to know every embedded type. The module encodes an `Any` as a tuple of type-URL plus raw base64 payload so it can be transparently relayed to clients. Source: [annotations-processor/README.md](annotations-processor/README.md) notes that the protobuf schema itself is generated from `@ProtoMessage`/`@ProtoField` annotations, keeping the binary and JSON representations in lockstep. ## Configuration and Persistence Touchpoints Although the engine itself is storage-agnostic, the runtime data (`Workflow`, `Task`) lands in a configurable indexing backend. Source: [es8-persistence/README.md](es8-persistence/README.md) describes the ES8 module that stores `${indexPrefix}_workflow`, `${indexPrefix}_task`, `${indexPrefix}_task_log`, `${indexPrefix}_message`, and `${indexPrefix}_event` indices via write aliases and an ILM policy. Community issues #226 and #928 highlight two operational pitfalls: ES7 persistence creating unbounded time-based indices when rollover is misconfigured, and ES7 Docker images that omit `conductor.indexing.type=elasticsearch` so Spring fails to wire the `IndexDAO` bean — both underscore the importance of matching indexing type to the selected persistence module. Source: [README.md](README.md) lists the supported backend combinations in `docker/server/config/`, including Redis+ES7/ES8/OpenSearch, Postgres, Postgres+ES7, and MySQL+ES7. ## See Also - [Workflow Definition & Task Definition Schemas](schemas/README.md) — full JSON Schema reference for `WorkflowDef`, `TaskDef`, `Workflow`, `Task` - [Community-contributed System Tasks](kafka/README.md) — Kafka, JQ, and other community task implementations - [AI Workflow Examples](ai/examples/README.md) — LLM and agentic task patterns - [Scheduler Examples](scheduler/examples/README.md) — cron-style triggering of workflow executions - [ES8 Persistence](es8-persistence/README.md) — how runtime `Workflow`/`Task` documents are stored and rolled over --- ## AI, LLM, MCP & Vector Database Integration ### Related Pages Related topics: [Workflow Engine, Tasks & Execution Model](#page-2)
Related Source Files The following source files were used to generate this page: - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java) - [ai/examples/README.md](https://github.com/conductor-oss/conductor/blob/main/ai/examples/README.md) - [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) - [ui-next/README.md](https://github.com/conductor-oss/conductor/blob/main/ui-next/README.md) - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) - [common/src/main/java/com/netflix/conductor/common/metadata/SchemaDef.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/SchemaDef.java) - [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md)
# AI, LLM, MCP & Vector Database Integration ## Overview Conductor exposes AI capabilities as first-class system task types so that language models, embedding stores, and external tool servers can be orchestrated the same way as any HTTP or business task. Workflow authors declare prompts, model parameters, vector-store identifiers, and MCP server URLs in JSON; the Conductor runtime handles scheduling, retry, persistence, and observability for each step. Source: [ai/examples/README.md:1-30]() This design is part of the project's "ship agents, not framework code" positioning: workers remain plain code in any language, while LLM-driven steps are declarative and machine-readable, which lets LLMs themselves generate and compose workflows. Source: [README.md:35-50]() ## LLM, Embedding, and MCP Task Types The catalog of AI-related task types is defined as enum constants and string constants in the shared metadata module: | Task Type | Constant String | Purpose | |---|---|---| | Text completion | `LLM_TEXT_COMPLETE` | Single-prompt text generation | | Chat completion | `LLM_CHAT_COMPLETE` | Multi-turn chat with system / user / assistant messages | | Index text | `LLM_INDEX_TEXT` | Push document chunks into a vector index | | Search index | `LLM_SEARCH_INDEX` | Similarity / RAG search over a vector index | | Generate embeddings | `LLM_GENERATE_EMBEDDINGS` | Produce vector embeddings from text | | Store embeddings | `LLM_STORE_EMBEDDINGS` | Persist pre-computed embeddings | | Get embeddings | `LLM_GET_EMBEDDINGS` | Retrieve stored embeddings by reference | | List MCP tools | `LIST_MCP_TOOLS` | Discover tools exposed by an MCP server | | Call MCP tool | `CALL_MCP_TOOL` | Invoke a discovered tool on an MCP server | | Pull workflow messages | `PULL_WORKFLOW_MESSAGES` | Drain queued chat messages for a workflow | Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java:1-30]() Because these are regular `Task` instances, the same runtime semantics apply: scheduled/started/updated timestamps, retry policies, queue-wait accounting, and worker attribution. Source: [common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java:30-50]() ## Provider and Pattern Examples The `ai/examples/` directory ships 22 reference workflows demonstrating the supported providers and orchestration patterns: - **Chat and reasoning** — `01-chat-completion.json` (OpenAI, Anthropic, Gemini, Vertex AI), `20-extended-thinking.json` (Anthropic with a thinking-token budget), `22-multi-turn-chain.json` (OpenAI `previousResponseId` chaining). - **Embeddings and RAG** — `02-embeddings.json`, `07-rag-complete.json` (OpenAI embeddings with PostgreSQL as the vector store). - **Tool and agent patterns** — `08-mcp-list-tools.json`, `09-mcp-call-tool.json` (weather), `10-mcp-ai-agent.json` (LLM + MCP). - **Media generation** — `11-video-openai-sora.json` (async), `12-video-gemini-veo.json` (async), `13-image-to-video-pipeline.json`, `14-stabilityai-image.json` (SD3.5). - **Document and research** — `15-pdf-generation.json`, `16-llm-to-pdf-pipeline.json`, `21-web-search-research-agent.json` (web search → synthesize → PDF). - **Specialized** — `17-web-search.json`, `18-code-execution.json` (Gemini sandbox), `19-coding-agent.json` (plan → write & run code → review). Source: [ai/examples/README.md:1-18]() The README's flagship pattern is an autonomous "think-act" agent: a `LIST_MCP_TOOLS` task discovers available tools, a `DO_WHILE` task alternates a `LLM_CHAT_COMPLETE` ("think") step with a tool invocation, and the loop exits when the model emits `done: true`. The MCP server URL is passed in as `${workflow.input.mcpServerUrl}` and the discovered tool list is injected into the system prompt as `${discover.output.tools}`. Source: [README.md:40-65]() ## MCP, Vector Stores, and Schemas ### Model Context Protocol (MCP) MCP integration is modeled as two task types — `LIST_MCP_TOOLS` and `CALL_MCP_TOOL` — so an external MCP server is just another declarative step in the workflow. This keeps the agent loop in the same JSON definition file as every other task, rather than requiring a custom worker. Source: [README.md:40-65]() ### Vector Database Operations Vector storage and retrieval are decomposed into three operations: `LLM_GENERATE_EMBEDDINGS` produces vectors, `LLM_INDEX_TEXT` writes them, and `LLM_SEARCH_INDEX` queries them. Example `07-rag-complete.json` wires all three together with OpenAI embeddings and PostgreSQL as the backing store. Source: [ai/examples/README.md:5-12]() ### Schema and Input Validation LLM task payloads are JSON objects (for example `messages: [...]`, `tools: [...]`) and can be guarded by a `SchemaDef` that supports inline JSON, AVRO, Protobuf, or an external registry reference via `externalRef`. Source: [common/src/main/java/com/netflix/conductor/common/metadata/SchemaDef.java:1-15]() The repository additionally publishes JSON Schemas (Draft 07) for the core models so editors can validate workflow and task definitions before registration. Source: [schemas/README.md:1-20]() ## Using the AI Tasks A typical lifecycle is identical to any Conductor workflow — register the definition, start an execution, then poll for the result: ```bash # 1. Register the workflow definition curl -X POST 'http://localhost:8080/api/metadata/workflow' \ -H 'Content-Type: application/json' \ -d @01-chat-completion.json # 2. Start an execution curl -X POST 'http://localhost:8080/api/workflow/chat_workflow' \ -H 'Content-Type: application/json' \ -d '{}' # 3. Check execution status curl -X GET 'http://localhost:8080/api/workflow/{workflowId}' ``` Source: [ai/examples/README.md:18-30]() The new `ui-next` console (React + Vite, served at `http://localhost:1234` in dev mode) provides the front-end for inspecting these workflows, including executions that mix LLM, MCP, and vector-store tasks. Source: [ui-next/README.md:1-20]() ## Common Failure Modes - **Async media generation** — `11-video-openai-sora.json` and `12-video-gemini-veo.json` are explicitly asynchronous; downstream tasks must poll for completion rather than expecting a synchronous result. Source: [ai/examples/README.md:8-10]() - **Provider configuration** — API keys, model identifiers, and (for Vertex AI) region are supplied at runtime; missing credentials surface as task failures rather than startup errors. - **MCP server reachability** — `LIST_MCP_TOOLS` is the entry point of the agent loop, so an unreachable MCP server fails the discover step and halts the whole workflow, which is also why it doubles as a connectivity check. - **Schema drift** — Providers evolve their message and tool-calling schemas; updates should be reflected in the corresponding JSON Schema files before re-registering definitions. Source: [schemas/README.md:1-20]() ## See Also - Workflow and Task JSON Schemas — [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md) - Task runtime model and queue-wait accounting — [Task.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java) - Build & run the server and UI — [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) --- ## Deployment, Configuration & Troubleshooting ### Related Pages Related topics: [Conductor Overview & System Architecture](#page-1), [Workflow Engine, Tasks & Execution Model](#page-2)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) - [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md) - [ui/package.json](https://github.com/conductor-oss/conductor/blob/main/ui/package.json) - [ui-next/package.json](https://github.com/conductor-oss/conductor/blob/main/ui-next/package.json) - [common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java) - [postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java](https://github.com/conductor-oss/conductor/blob/main/postgres-persistence/src/main/java/com/netflix/conductor/postgres/dao/PostgresMetadataDAO.java) - [common/src/main/java/com/netflix/conductor/common/metadata/workflow/StateChangeEvent.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/workflow/StateChangeEvent.java) - [annotations-processor/README.md](https://github.com/conductor-oss/conductor/blob/main/annotations-processor/README.md) - [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md) - [kafka/README.md](https://github.com/conductor-oss/conductor/blob/main/kafka/README.md)
# Deployment, Configuration & Troubleshooting ## Overview Conductor OSS is a workflow orchestration engine that ships as a multi-module Gradle project. The server (a Spring Boot application) is paired with pluggable persistence backends: a primary datastore (Redis, Postgres, MySQL, or Cassandra) for queue and runtime state, and an indexing backend (Elasticsearch 7, Elasticsearch 8, or OpenSearch) for search and visibility. The current release line is **v3.31.0-rc.4** as listed in [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md). Deployment, configuration, and troubleshooting are tightly coupled: a successful installation depends on selecting the correct backend module, providing the right indexing type property, and matching the JDK version expected by the build. ## Deployment Topology The default Docker Compose stack brings up the conductor server, a database, an index store, and the new `ui-next` console, all wired through Spring Boot auto-configuration. ```mermaid flowchart LR Worker[External Workers] -->|HTTP poll / task ack| Server[Conductor Server] UI[ui-next React UI] -->|REST| Server CLI[Conductor CLI / curl] -->|REST| Server Server -->|queue + state| DB[(Redis / Postgres / MySQL / Cassandra)] Server -->|index workflow + task| IDX[(Elasticsearch 7 / 8 / OpenSearch)] Server -.->|annotation processor| AP[protobuf generation] ``` The `ui-next` workspace replaces the legacy `ui` in published releases; both are still present in the repository, but recent publish pipelines now point at `ui-next` (see the v3.31.0-rc.4 changelog referencing PR #1168 in [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md)). ### Backend Configurations Six pre-bundled property files live in `docker/server/config/` and are selected at container startup. The choice of file determines both the primary datastore and the indexing backend, and the indexing type property must match the file in use. | Backend combination | Property file | |---------------------|---------------| | Redis + Elasticsearch 7 (default) | `config-redis.properties` | | Redis + Elasticsearch 8 | `config-redis-es8.properties` | | Redis + OpenSearch | `config-redis-os.properties` | | Postgres (no ES) | `config-postgres.properties` | | Postgres + Elasticsearch 7 | `config-postgres-es7.properties` | | MySQL + Elasticsearch 7 | `config-mysql.properties` | Source: [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) (Backend Configuration table). ## Configuration ### Indexing Backend Selection The `conductor.indexing.type` property is the single switch that activates the correct `IndexDAO` bean. Setting it to `elasticsearch` activates the ES7 module, `elasticsearch8` activates the ES8 module, and `opensearch` activates the OpenSearch integration. The ES8 module additionally uses composable index templates, write aliases, and an ILM policy (`max_primary_shard_size=50gb`) backed by the `elasticsearch-java` 8.19.x client. Source: [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md). When building from source the matching Gradle module must be selected to avoid Lucene conflicts: ```sh ./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8 ``` `-PindexingBackend=es8` is accepted as a shortcut. Source: [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md). ### Build and Runtime Requirements Build from source requires Docker Desktop, JDK 21+, and Node.js 18+ with pnpm for the UI. Source: [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) (Build From Source section). The legacy `ui` package declares `node >=14.17.0` while `ui-next` uses a peerDependencies list aligned with React 18 and MUI v7. Sources: [ui/package.json](https://github.com/conductor-oss/conductor/blob/main/ui/package.json) and [ui-next/package.json](https://github.com/conductor-oss/conductor/blob/main/ui-next/package.json). ### Annotation Processing Custom task and workflow definitions compiled against the project go through the `annotations-processor` module, which generates protobuf descriptors from `@ProtoMessage`/`@ProtoField` annotations. The generated wire format is what enables cross-language gRPC clients to interoperate with the server. Source: [annotations-processor/README.md](https://github.com/conductor-oss/conductor/blob/main/annotations-processor/README.md). ### Schema Validation JSON Schemas under `schemas/` mirror the Java model classes and can be used by IDEs (VS Code, IntelliJ) for editor-time validation of workflow and task definitions. They conform to **JSON Schema Draft 07** and support `$ref` for composition, mirroring the recursive `WorkflowTask` structure. Source: [schemas/README.md](https://github.com/conductor-oss/conductor/blob/main/schemas/README.md). ## Common Issues and Troubleshooting ### 1. Server fails to start: "no IndexDAO bean" The most common startup failure, tracked in issue #928, is an absent `conductor.indexing.type=elasticsearch` line in the ES7 Docker config. Without it, Spring does not instantiate the ES7 `IndexDAO` bean and bootstrap aborts. **Fix:** explicitly set the indexing type in the active properties file to match the bundled config (e.g. `elasticsearch`, `elasticsearch8`, or `opensearch`). Source: [README.md](https://github.com/conductor-oss/conductor/blob/main/README.md) (Backend Configuration). ### 2. JDK mismatch in Docker image Issue #711 reports that `docker/server/Dockerfile` references Java 17 while `build.gradle` requires Java 21 (`sourceCompatibility = JavaVersion.VERSION_21`). The runtime image must be rebuilt against a JDK 21 base image to match the toolchain; otherwise the boot jar will not execute. ### 3. Parent workflow hangs after a sub-workflow completes Issue #712 describes a regression in v3.21.23 Docker images where the parent workflow stops progressing after the child `SUB_WORKFLOW` task completes. The parent can be unstuck by issuing `POST /api/workflow/decide/{workflowId}`. Sub-workflow references are tracked via the `subWorkflowParam` field and the `SUB_WORKFLOW` task type constant declared in [TaskType.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java). ### 4. Unbounded index growth in ES7 persistence Issue #226 documents an older `ElasticSearchRestDAOV7` path that creates a new `conductor_task_log_YYYYMMDD` index daily without rollover, leading to thousands of shards. The modern ES8 module fixes this by using composable index templates plus an ILM policy capped at 50gb primary shard size. **Recommendation:** migrate to the ES8 backend, or configure a manual rollover/ILM policy on the ES7 indices. Source: [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md). ### 5. ES8 compatibility Long-running request #49 asking for ES8 support is now resolved via the dedicated `es8-persistence` module, which uses the official `elasticsearch-java` 8.19.11 client and ships with a pinned `8.19.x` Docker image. Source: [es8-persistence/README.md](https://github.com/conductor-oss/conductor/blob/main/es8-persistence/README.md). ### 6. Duplicate task scheduling (attempt 0 × 2) A new FAQ entry (added in v3.31.0-rc.4 per PR #1173) addresses a duplicate scheduling symptom where the same task appears twice at attempt 0. The typical cause is a worker acking the task before the queue listener has finished writing the SCHEDULED record; polling workers should treat the first `GET /tasks` response as authoritative and use idempotency keys on the ack call. ## Operational Tips - Always set `conductor.indexing.type` explicitly — relying on defaults hides backend selection errors until the first indexing call. - Pair the Gradle build flag (`-PindexingBackend=…`) with the runtime property; mismatches produce classpath duplicates from Lucene. - Use `WorkflowTestRequest` to mock sub-workflow outputs during integration tests so the parent/child completion path can be validated without spinning up real workers. Source: [common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java). - For state-change observability, subscribe to `StateChangeEvent` payloads rather than polling task status. Source: [common/src/main/java/com/netflix/conductor/common/metadata/workflow/StateChangeEvent.java](https://github.com/conductor-oss/conductor/blob/main/common/src/main/java/com/netflix/conductor/common/metadata/workflow/StateChangeEvent.java). ## See Also - Architecture overview - Task types reference - Workflow definition schema - Elasticsearch backend modules (ES7, ES8, OpenSearch) --- --- ## Pitfall Log Project: conductor-oss/conductor Summary: Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification. ## 1. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/conductor-oss/conductor ## 2. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/conductor-oss/conductor/issues/396 ## 3. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor ## 4. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/conductor-oss/conductor ## 5. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/conductor-oss/conductor ## 6. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor ## 7. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor