Doramagic Project Pack · Human Manual
conductor
Conductor is an event driven agentic workflow engine providing durable and highly resilient execution engine for applications and AI Agents
Conductor Overview & System Architecture
Related topics: Workflow Engine, Tasks & Execution Model, Deployment, Configuration & Troubleshooting
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Workflow Engine, Tasks & Execution Model, Deployment, Configuration & Troubleshooting
Conductor Overview & System Architecture
1. Purpose and Scope
Conductor is an open-source workflow orchestration engine originally created at Netflix and now maintained as the conductor-oss project under the Apache 2.0 license. The top-level README.md:1-30 positions Conductor as a platform for authoring, executing, and observing durable, stateful workflows that coordinate heterogeneous tasks (HTTP calls, microservices, AI agents, scripts, message-queue work, and more). The repository ships:
- A server module exposing REST APIs and the orchestration runtime.
- A common module holding shared data models used by both server and client SDKs.
- Pluggable persistence modules — one of which is the Postgres-backed implementation in
postgres-persistence/. - Optional community-contributed integrations such as the Kafka system tasks described in kafka/README.md:1-15.
- A growing set of AI workflow examples (LLM chat, code agents, research agents) under ai/examples/README.md:1-25 that illustrate how to register and execute LLM-driven workflows through the same REST surface.
A separate project, conductor-oss/rust-sdk, provides an incubating Rust client SDK, reflecting the project's multi-language direction.
2. High-Level Architecture
Conductor follows a layered, interface-driven design. The orchestration engine is decoupled from any specific datastore, and persistence is selected at boot time through Spring conditional configuration.
flowchart TB
Client["Client / SDK / UI"] -->|REST + JSON| Server["Conductor Server<br/>(Spring Boot)"]
Server --> Core["Core Orchestration<br/>(WorkflowExecutor, DeciderService,<br/>SystemTask workers)"]
Core --> DAO["DAO Interfaces<br/>(MetadataDAO, ExecutionDAO,<br/>QueueDAO, IndexDAO)"]
DAO --> Pg["Postgres DAO<br/>(PostgresMetadataDAO,<br/>PostgresExecutionDAO,<br/>PostgresQueueDAO,<br/>PostgresIndexDAO)"]
Pg --> Db[("PostgreSQL")]
Core --> Async["Async Queues<br/>(PostgresQueueDAO listener)"]
Async --> PgThe common module defines the public-facing data models — WorkflowDef, TaskDef, Workflow, Task, TaskResult, and friends — and ships JSON Schemas for client-side validation. Source: schemas/README.md:1-30. The recursive WorkflowTask schema enables unlimited nesting of decision, fork-join, do-while, and sub-workflow patterns. The same models are surfaced through Protobuf annotations (e.g. @ProtoMessage / @ProtoField) on classes such as CacheConfig.java:13-22 and ExecutionMetadata.java:19-26, which is why the runtime can support both JSON REST and binary gRPC transports from a single source of truth.
Pluggable persistence
The conductor.db.type Spring property gates which DAO implementation is wired in. For Postgres, see the conditional bean in PostgresConfiguration.java:42-66 (@ConditionalOnProperty(name = "conductor.db.type", havingValue = "postgres")), and the toggleable properties in PostgresProperties.java:18-44 (task-def cache refresh, deadlock retry max, queue-notify toggles, data-migration flag). The README enumerates the supported backend combinations in README.md:1-50: Redis with ES7/ES8/OpenSearch, and Postgres-only or Postgres-plus-ES variants.
3. Persistence Layer (Postgres Implementation)
The Postgres persistence module implements the four core DAO contracts through one shared base class and four concrete DAOs.
- PostgresBaseDAO.java:36-70 — supplies the shared
DataSource,ObjectMapper, and aRetryTemplateused for deadlock/connection retries; it also filters out the DAO's own stack frames from logged exceptions. - PostgresMetadataDAO.java:1-40 — stores
WorkflowDef,TaskDef, andEventHandlerdefinitions, with an in-memory cache refreshed everytaskDefCacheRefreshIntervalseconds (default 60s). - PostgresExecutionDAO.java:1-40 — persists workflow and task state, plus the rate-limiting and concurrent-execution limit tables; it pushes events through
Monitorsfor metrics. - PostgresQueueDAO.java:1-40 — implements task queues; uses a
PostgresQueueListenerfor in-process notifications whenexperimentalQueueNotify=trueand unacks stalled messages after 60s. - PostgresIndexDAO.java:1-40 — provides searchable indices of workflows, tasks, and
TaskExecLogentries without requiring an external search engine.
Community note: the analogous Elasticsearch 7 path has a long-standing complaint (issue #226) thatcreateIndexesTemplatesproduced an unbounded number ofconductor_task_log_YYYYMMDDindices, eventually exhausting shards. The PostgresIndexDAOsidesteps that by storing the log in relational tables.
4. Core Data Models & Execution Surface
The common module defines the contracts that flow over the wire. Beyond the legacy Workflow / Task shapes, several newer models are worth knowing:
- WorkflowSummaryExtended.java:18-40 — extends
WorkflowSummaryso thatinputandoutputare returned asMap<String,Object>for JSON consumers, while staying Protobuf-compatible via dedicated field ids. - WorkflowRun.java:1-25 and TaskRun.java:1-30 — DTOs used by the signal-handling endpoints to return *what is currently blocking* a workflow or task. Both extend the abstract SignalResponse.java:15-30, which carries
workflowId,correlationId,requestId, and the chosen return strategy. - WorkflowSignalReturnStrategy.java:8-30 — enum selecting between
TARGET_WORKFLOW,BLOCKING_WORKFLOW,BLOCKING_TASK, andBLOCKING_TASK_INPUT. This is the key control for how deep into nested sub-workflows a signal response should be materialised. The accompanying comment notes this was a non-trivial design point once the notification service was introduced. - WorkflowTestRequest.java:1-40 — request body for the workflow *test* endpoint; supports per-task mocks (
taskRefToMockOutput) and per-sub-workflow mocks (subWorkflowTestRequest), plus simulatedexecutionTimeandqueueWaitTimefor timeout tests. - CacheConfig.java:13-40 — per-workflow response-cache configuration (
key+ttlInSecond).
5. Build, Runtime, and Operational Gotchas
Per README.md:1-60, building from source requires JDK 21+, Docker Desktop, Node.js 18+, and pnpm (for the UI). Community issue #711 highlights that the bundled docker/server/Dockerfile historically referenced Java 17 in two places, which conflicts with the JavaLanguageVersion.of(21) toolchain declared in build.gradle. Operators upgrading from older Conductor 3.21.x should also be aware of issue #712, in which parent workflows can hang after a sub-workflow completes and require a manual POST /api/workflow/decide/{id} to unblock.
Elasticsearch users should set conductor.indexing.type=elasticsearch explicitly (issue #928) so that the ES7 IndexDAO bean is registered; without it Spring's auto-configuration can fail to start the server. For ES8 (issue #49), use the dedicated config-redis-es8.properties profile.
The latest release line (v3.31.0-rc.4) ships the new ui-next front-end, an updated FAQ entry for duplicate task scheduling on attempt 0, and test-harness fixes — see the release notes for the full change set.
See Also
- Schemas Reference — JSON Schemas for
WorkflowDefandTaskDef - Postgres Persistence Configuration — tuning guide for
PostgresProperties - Signal API &
WorkflowSignalReturnStrategy— deep-dive on the signal response models - Kafka Community Tasks — system tasks contributed under the
conductor-taskartifact - AI Workflow Examples — end-to-end LLM-agent workflow samples
Source: https://github.com/conductor-oss/conductor / Human Manual
Workflow Engine, Tasks & Execution Model
Related topics: Conductor Overview & System Architecture, AI, LLM, MCP & Vector Database Integration
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Conductor Overview & System Architecture, AI, LLM, MCP & Vector Database Integration
Workflow Engine, Tasks & Execution Model
Purpose and Scope
Conductor's workflow engine orchestrates long-running, distributed business processes by interpreting declarative workflow definitions and driving tasks through a well-defined lifecycle. A workflow definition (the *blueprint*) is registered once and used to spin off many runtime workflow *instances*. Each instance schedules and tracks the tasks defined inside it, dispatching them to external workers (HTTP, gRPC, polling), system tasks (Kafka, JQ, HTTP, LLM), or built-in control-flow tasks (DECISION, SWITCH, FORK_JOIN, DO_WHILE, SUB_WORKFLOW).
Source: schemas/README.md describes the four core schemas — WorkflowDef, TaskDef, Workflow, and Task — that model this two-layer split between template and runtime.
Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java enumerates every built-in task kind Conductor knows how to interpret, including simple, control-flow, integration, and LLM/MCP tasks.
Core Data Model
Definitions vs. Instances
| Schema / Class | Role | Notes |
|---|---|---|
WorkflowDef | Blueprint of a workflow | Recursive WorkflowTask allows unlimited nesting of DECISION/SWITCH/FORK_JOIN/DO_WHILE/SUB_WORKFLOW Source: schemas/README.md |
TaskDef | Reusable task template | Carries retry count/delay, timeout policy (RETRY/TIME_OUT_WF/ALERT_ONLY), and rate-limit settings Source: schemas/README.md |
Workflow | Live runtime instance | Carries status, task list, history (recursive for versioning), priority 0–99, parent/child links Source: schemas/README.md |
Task | Live runtime task | Carries status, input/output, workerId, retry counters, iteration, subWorkflowId Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java |
Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java shows Task uses @ProtoField annotations for compact binary serialization while remaining compatible with JSON via the proto-aware Jackson module.
Task Statuses
Source: schemas/README.md defines the runtime state machine:
- In-flight:
SCHEDULED,IN_PROGRESS - Success:
COMPLETED,COMPLETED_WITH_ERRORS,SKIPPED - Failure:
FAILED,FAILED_WITH_TERMINAL_ERROR,TIMED_OUT,CANCELED
Workflow-level statuses are RUNNING, COMPLETED, FAILED, TIMED_OUT, TERMINATED, PAUSED.
Task Types and Execution
Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java enumerates every supported task kind. The enum is mirrored as string constants (e.g. TASK_TYPE_HTTP = "HTTP") so call sites never hardcode literals. Categories present in the enum include:
| Category | Examples |
|---|---|
| Simple / Worker | SIMPLE, HTTP, INLINE, LAMBDA |
| Control-flow | DECISION, SWITCH, JOIN, EXCLUSIVE_JOIN, FORK_JOIN, FORK_JOIN_DYNAMIC, DO_WHILE, EVENT, WAIT, HUMAN |
| Sub-workflows | SUB_WORKFLOW, START_WORKFLOW |
| AI / LLM | LLM_TEXT_COMPLETE, LLM_CHAT_COMPLETE, LLM_INDEX_TEXT, LLM_SEARCH_INDEX, LLM_GENERATE_EMBEDDINGS, LLM_STORE_EMBEDDINGS, LLM_GET_EMBEDDINGS |
| MCP / agentic | LIST_MCP_TOOLS, CALL_MCP_TOOL, PULL_WORKFLOW_MESSAGES |
Source: kafka/README.md shows how community-contributed system tasks like KAFKA_PUBLISH and JSON_JQ_TRANSFORM_TASK are modeled — a task declares its type and an inputParameters map. JQ tasks, for example, must include a queryExpression plus any user-named variables.
Source: ai/examples/README.md illustrates end-to-end execution: a workflow such as chat_workflow is registered via POST /api/metadata/workflow, started via POST /api/workflow/{name}, and inspected via GET /api/workflow/{workflowId}. Inputs and outputs are passed as JSON objects, and outputs from earlier tasks are referenced by ${taskRefName.output.field} placeholders.
Execution Flow
flowchart LR
A[WorkflowDef registered] --> B[POST /api/workflow/name]
B --> C[Workflow instance created]
C --> D[Engine maps WorkflowTasks -> Tasks]
D --> E{Task type}
E -->|Simple/HTTP| F[Worker polls & completes]
E -->|System task| G[In-process executor<br/>HTTP, JQ, Kafka, LLM]
E -->|Control-flow| H[Branch / fork / loop / sub-workflow]
F & G & H --> I[Task status updated]
I --> J{Workflow complete?}
J -->|No| D
J -->|Yes| K[Workflow terminal status]Source: schemas/README.md confirms that nested control-flow tasks — decisionCases, defaultCase, forkTasks, loopOver, loopCondition, subWorkflowParam — are encoded directly in the recursive WorkflowTask definition.
Source: common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java shows how events plug into execution: an EventHandler carries an actions list and a nested TaskDetails message that references the producing workflowId, taskRefName, taskId, and output, enabling event-driven task completion without polling.
Sub-Workflow and Async Patterns
Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java reserves fields for sub-workflow coordination: subWorkflowId and workflowInstanceId link a SUB_WORKFLOW task to the child workflow it spawned, while workflowPriority lets high-priority runs preempt lower ones.
A common community-reported failure mode (referenced in issue #712 *"The workflow hangs after running Sub workflow"*) is that the parent workflow does not automatically resume after the child completes until an external POST /api/workflow/decide/{workflowId} is issued. This is consistent with the explicit linking fields in Task and emphasizes that sub-workflow completion must be observed through the normal async-decision loop.
Serialization and Wire Format
Source: common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java explains how Any protobufs (used for inputMessage and outputMessage on Task, and outputMessage on EventHandler.TaskDetails) are exposed over REST without requiring the server to know every embedded type. The module encodes an Any as a tuple of type-URL plus raw base64 payload so it can be transparently relayed to clients.
Source: annotations-processor/README.md notes that the protobuf schema itself is generated from @ProtoMessage/@ProtoField annotations, keeping the binary and JSON representations in lockstep.
Configuration and Persistence Touchpoints
Although the engine itself is storage-agnostic, the runtime data (Workflow, Task) lands in a configurable indexing backend. Source: es8-persistence/README.md describes the ES8 module that stores ${indexPrefix}_workflow, ${indexPrefix}_task, ${indexPrefix}_task_log, ${indexPrefix}_message, and ${indexPrefix}_event indices via write aliases and an ILM policy. Community issues #226 and #928 highlight two operational pitfalls: ES7 persistence creating unbounded time-based indices when rollover is misconfigured, and ES7 Docker images that omit conductor.indexing.type=elasticsearch so Spring fails to wire the IndexDAO bean — both underscore the importance of matching indexing type to the selected persistence module.
Source: README.md lists the supported backend combinations in docker/server/config/, including Redis+ES7/ES8/OpenSearch, Postgres, Postgres+ES7, and MySQL+ES7.
See Also
- Workflow Definition & Task Definition Schemas — full JSON Schema reference for
WorkflowDef,TaskDef,Workflow,Task - Community-contributed System Tasks — Kafka, JQ, and other community task implementations
- AI Workflow Examples — LLM and agentic task patterns
- Scheduler Examples — cron-style triggering of workflow executions
- ES8 Persistence — how runtime
Workflow/Taskdocuments are stored and rolled over
Source: https://github.com/conductor-oss/conductor / Human Manual
AI, LLM, MCP & Vector Database Integration
Related topics: Workflow Engine, Tasks & Execution Model
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Workflow Engine, Tasks & Execution Model
AI, LLM, MCP & Vector Database Integration
Overview
Conductor exposes AI capabilities as first-class system task types so that language models, embedding stores, and external tool servers can be orchestrated the same way as any HTTP or business task. Workflow authors declare prompts, model parameters, vector-store identifiers, and MCP server URLs in JSON; the Conductor runtime handles scheduling, retry, persistence, and observability for each step. Source: ai/examples/README.md:1-30
This design is part of the project's "ship agents, not framework code" positioning: workers remain plain code in any language, while LLM-driven steps are declarative and machine-readable, which lets LLMs themselves generate and compose workflows. Source: README.md:35-50
LLM, Embedding, and MCP Task Types
The catalog of AI-related task types is defined as enum constants and string constants in the shared metadata module:
| Task Type | Constant String | Purpose |
|---|---|---|
| Text completion | LLM_TEXT_COMPLETE | Single-prompt text generation |
| Chat completion | LLM_CHAT_COMPLETE | Multi-turn chat with system / user / assistant messages |
| Index text | LLM_INDEX_TEXT | Push document chunks into a vector index |
| Search index | LLM_SEARCH_INDEX | Similarity / RAG search over a vector index |
| Generate embeddings | LLM_GENERATE_EMBEDDINGS | Produce vector embeddings from text |
| Store embeddings | LLM_STORE_EMBEDDINGS | Persist pre-computed embeddings |
| Get embeddings | LLM_GET_EMBEDDINGS | Retrieve stored embeddings by reference |
| List MCP tools | LIST_MCP_TOOLS | Discover tools exposed by an MCP server |
| Call MCP tool | CALL_MCP_TOOL | Invoke a discovered tool on an MCP server |
| Pull workflow messages | PULL_WORKFLOW_MESSAGES | Drain queued chat messages for a workflow |
Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java:1-30
Because these are regular Task instances, the same runtime semantics apply: scheduled/started/updated timestamps, retry policies, queue-wait accounting, and worker attribution. Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java:30-50
Provider and Pattern Examples
The ai/examples/ directory ships 22 reference workflows demonstrating the supported providers and orchestration patterns:
- Chat and reasoning —
01-chat-completion.json(OpenAI, Anthropic, Gemini, Vertex AI),20-extended-thinking.json(Anthropic with a thinking-token budget),22-multi-turn-chain.json(OpenAIpreviousResponseIdchaining). - Embeddings and RAG —
02-embeddings.json,07-rag-complete.json(OpenAI embeddings with PostgreSQL as the vector store). - Tool and agent patterns —
08-mcp-list-tools.json,09-mcp-call-tool.json(weather),10-mcp-ai-agent.json(LLM + MCP). - Media generation —
11-video-openai-sora.json(async),12-video-gemini-veo.json(async),13-image-to-video-pipeline.json,14-stabilityai-image.json(SD3.5). - Document and research —
15-pdf-generation.json,16-llm-to-pdf-pipeline.json,21-web-search-research-agent.json(web search → synthesize → PDF). - Specialized —
17-web-search.json,18-code-execution.json(Gemini sandbox),19-coding-agent.json(plan → write & run code → review).
Source: ai/examples/README.md:1-18
The README's flagship pattern is an autonomous "think-act" agent: a LIST_MCP_TOOLS task discovers available tools, a DO_WHILE task alternates a LLM_CHAT_COMPLETE ("think") step with a tool invocation, and the loop exits when the model emits done: true. The MCP server URL is passed in as ${workflow.input.mcpServerUrl} and the discovered tool list is injected into the system prompt as ${discover.output.tools}. Source: README.md:40-65
MCP, Vector Stores, and Schemas
Model Context Protocol (MCP)
MCP integration is modeled as two task types — LIST_MCP_TOOLS and CALL_MCP_TOOL — so an external MCP server is just another declarative step in the workflow. This keeps the agent loop in the same JSON definition file as every other task, rather than requiring a custom worker. Source: README.md:40-65
Vector Database Operations
Vector storage and retrieval are decomposed into three operations: LLM_GENERATE_EMBEDDINGS produces vectors, LLM_INDEX_TEXT writes them, and LLM_SEARCH_INDEX queries them. Example 07-rag-complete.json wires all three together with OpenAI embeddings and PostgreSQL as the backing store. Source: ai/examples/README.md:5-12
Schema and Input Validation
LLM task payloads are JSON objects (for example messages: [...], tools: [...]) and can be guarded by a SchemaDef that supports inline JSON, AVRO, Protobuf, or an external registry reference via externalRef. Source: common/src/main/java/com/netflix/conductor/common/metadata/SchemaDef.java:1-15 The repository additionally publishes JSON Schemas (Draft 07) for the core models so editors can validate workflow and task definitions before registration. Source: schemas/README.md:1-20
Using the AI Tasks
A typical lifecycle is identical to any Conductor workflow — register the definition, start an execution, then poll for the result:
# 1. Register the workflow definition
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
-H 'Content-Type: application/json' \
-d @01-chat-completion.json
# 2. Start an execution
curl -X POST 'http://localhost:8080/api/workflow/chat_workflow' \
-H 'Content-Type: application/json' \
-d '{}'
# 3. Check execution status
curl -X GET 'http://localhost:8080/api/workflow/{workflowId}'
Source: ai/examples/README.md:18-30
The new ui-next console (React + Vite, served at http://localhost:1234 in dev mode) provides the front-end for inspecting these workflows, including executions that mix LLM, MCP, and vector-store tasks. Source: ui-next/README.md:1-20
Common Failure Modes
- Async media generation —
11-video-openai-sora.jsonand12-video-gemini-veo.jsonare explicitly asynchronous; downstream tasks must poll for completion rather than expecting a synchronous result. Source: ai/examples/README.md:8-10 - Provider configuration — API keys, model identifiers, and (for Vertex AI) region are supplied at runtime; missing credentials surface as task failures rather than startup errors.
- MCP server reachability —
LIST_MCP_TOOLSis the entry point of the agent loop, so an unreachable MCP server fails the discover step and halts the whole workflow, which is also why it doubles as a connectivity check. - Schema drift — Providers evolve their message and tool-calling schemas; updates should be reflected in the corresponding JSON Schema files before re-registering definitions. Source: schemas/README.md:1-20
See Also
- Workflow and Task JSON Schemas — schemas/README.md
- Task runtime model and queue-wait accounting — Task.java
- Build & run the server and UI — README.md
Source: https://github.com/conductor-oss/conductor / Human Manual
Deployment, Configuration & Troubleshooting
Related topics: Conductor Overview & System Architecture, Workflow Engine, Tasks & Execution Model
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Continue reading this section for the full explanation and source context.
Related Pages
Related topics: Conductor Overview & System Architecture, Workflow Engine, Tasks & Execution Model
Deployment, Configuration & Troubleshooting
Overview
Conductor OSS is a workflow orchestration engine that ships as a multi-module Gradle project. The server (a Spring Boot application) is paired with pluggable persistence backends: a primary datastore (Redis, Postgres, MySQL, or Cassandra) for queue and runtime state, and an indexing backend (Elasticsearch 7, Elasticsearch 8, or OpenSearch) for search and visibility. The current release line is v3.31.0-rc.4 as listed in README.md. Deployment, configuration, and troubleshooting are tightly coupled: a successful installation depends on selecting the correct backend module, providing the right indexing type property, and matching the JDK version expected by the build.
Deployment Topology
The default Docker Compose stack brings up the conductor server, a database, an index store, and the new ui-next console, all wired through Spring Boot auto-configuration.
flowchart LR
Worker[External Workers] -->|HTTP poll / task ack| Server[Conductor Server]
UI[ui-next React UI] -->|REST| Server
CLI[Conductor CLI / curl] -->|REST| Server
Server -->|queue + state| DB[(Redis / Postgres / MySQL / Cassandra)]
Server -->|index workflow + task| IDX[(Elasticsearch 7 / 8 / OpenSearch)]
Server -.->|annotation processor| AP[protobuf generation]The ui-next workspace replaces the legacy ui in published releases; both are still present in the repository, but recent publish pipelines now point at ui-next (see the v3.31.0-rc.4 changelog referencing PR #1168 in README.md).
Backend Configurations
Six pre-bundled property files live in docker/server/config/ and are selected at container startup. The choice of file determines both the primary datastore and the indexing backend, and the indexing type property must match the file in use.
| Backend combination | Property file |
|---|---|
| Redis + Elasticsearch 7 (default) | config-redis.properties |
| Redis + Elasticsearch 8 | config-redis-es8.properties |
| Redis + OpenSearch | config-redis-os.properties |
| Postgres (no ES) | config-postgres.properties |
| Postgres + Elasticsearch 7 | config-postgres-es7.properties |
| MySQL + Elasticsearch 7 | config-mysql.properties |
Source: README.md (Backend Configuration table).
Configuration
Indexing Backend Selection
The conductor.indexing.type property is the single switch that activates the correct IndexDAO bean. Setting it to elasticsearch activates the ES7 module, elasticsearch8 activates the ES8 module, and opensearch activates the OpenSearch integration. The ES8 module additionally uses composable index templates, write aliases, and an ILM policy (max_primary_shard_size=50gb) backed by the elasticsearch-java 8.19.x client. Source: es8-persistence/README.md.
When building from source the matching Gradle module must be selected to avoid Lucene conflicts:
./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8
-PindexingBackend=es8 is accepted as a shortcut. Source: es8-persistence/README.md.
Build and Runtime Requirements
Build from source requires Docker Desktop, JDK 21+, and Node.js 18+ with pnpm for the UI. Source: README.md (Build From Source section). The legacy ui package declares node >=14.17.0 while ui-next uses a peerDependencies list aligned with React 18 and MUI v7. Sources: ui/package.json and ui-next/package.json.
Annotation Processing
Custom task and workflow definitions compiled against the project go through the annotations-processor module, which generates protobuf descriptors from @ProtoMessage/@ProtoField annotations. The generated wire format is what enables cross-language gRPC clients to interoperate with the server. Source: annotations-processor/README.md.
Schema Validation
JSON Schemas under schemas/ mirror the Java model classes and can be used by IDEs (VS Code, IntelliJ) for editor-time validation of workflow and task definitions. They conform to JSON Schema Draft 07 and support $ref for composition, mirroring the recursive WorkflowTask structure. Source: schemas/README.md.
Common Issues and Troubleshooting
1. Server fails to start: "no IndexDAO bean"
The most common startup failure, tracked in issue #928, is an absent conductor.indexing.type=elasticsearch line in the ES7 Docker config. Without it, Spring does not instantiate the ES7 IndexDAO bean and bootstrap aborts. Fix: explicitly set the indexing type in the active properties file to match the bundled config (e.g. elasticsearch, elasticsearch8, or opensearch). Source: README.md (Backend Configuration).
2. JDK mismatch in Docker image
Issue #711 reports that docker/server/Dockerfile references Java 17 while build.gradle requires Java 21 (sourceCompatibility = JavaVersion.VERSION_21). The runtime image must be rebuilt against a JDK 21 base image to match the toolchain; otherwise the boot jar will not execute.
3. Parent workflow hangs after a sub-workflow completes
Issue #712 describes a regression in v3.21.23 Docker images where the parent workflow stops progressing after the child SUB_WORKFLOW task completes. The parent can be unstuck by issuing POST /api/workflow/decide/{workflowId}. Sub-workflow references are tracked via the subWorkflowParam field and the SUB_WORKFLOW task type constant declared in TaskType.java.
4. Unbounded index growth in ES7 persistence
Issue #226 documents an older ElasticSearchRestDAOV7 path that creates a new conductor_task_log_YYYYMMDD index daily without rollover, leading to thousands of shards. The modern ES8 module fixes this by using composable index templates plus an ILM policy capped at 50gb primary shard size. Recommendation: migrate to the ES8 backend, or configure a manual rollover/ILM policy on the ES7 indices. Source: es8-persistence/README.md.
5. ES8 compatibility
Long-running request #49 asking for ES8 support is now resolved via the dedicated es8-persistence module, which uses the official elasticsearch-java 8.19.11 client and ships with a pinned 8.19.x Docker image. Source: es8-persistence/README.md.
6. Duplicate task scheduling (attempt 0 × 2)
A new FAQ entry (added in v3.31.0-rc.4 per PR #1173) addresses a duplicate scheduling symptom where the same task appears twice at attempt 0. The typical cause is a worker acking the task before the queue listener has finished writing the SCHEDULED record; polling workers should treat the first GET /tasks response as authoritative and use idempotency keys on the ack call.
Operational Tips
- Always set
conductor.indexing.typeexplicitly — relying on defaults hides backend selection errors until the first indexing call. - Pair the Gradle build flag (
-PindexingBackend=…) with the runtime property; mismatches produce classpath duplicates from Lucene. - Use
WorkflowTestRequestto mock sub-workflow outputs during integration tests so the parent/child completion path can be validated without spinning up real workers. Source: common/src/main/java/com/netflix/conductor/common/run/WorkflowTestRequest.java. - For state-change observability, subscribe to
StateChangeEventpayloads rather than polling task status. Source: common/src/main/java/com/netflix/conductor/common/metadata/workflow/StateChangeEvent.java.
See Also
- Architecture overview
- Task types reference
- Workflow definition schema
- Elasticsearch backend modules (ES7, ES8, OpenSearch)
Source: https://github.com/conductor-oss/conductor / Human Manual
Doramagic Pitfall Log
Source-linked risks stay visible on the manual page so the preview does not read like a recommendation.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
May increase setup, validation, or first-run risk for the user.
Doramagic Pitfall Log
Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification.
1. Capability evidence risk: Capability evidence risk requires verification
- Severity: medium
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: capability.assumptions | https://github.com/conductor-oss/conductor
2. Maintenance risk: Maintenance risk requires verification
- Severity: medium
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: community_evidence:github | https://github.com/conductor-oss/conductor/issues/396
3. Maintenance risk: Maintenance risk requires verification
- Severity: medium
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor
4. Security or permission risk: Security or permission risk requires verification
- Severity: medium
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: downstream_validation.risk_items | https://github.com/conductor-oss/conductor
5. Security or permission risk: Security or permission risk requires verification
- Severity: medium
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: risks.scoring_risks | https://github.com/conductor-oss/conductor
6. Maintenance risk: Maintenance risk requires verification
- Severity: low
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor
7. Maintenance risk: Maintenance risk requires verification
- Severity: low
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Recommended check: Reproduce the official install and quickstart path in an isolated environment.
- Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor
Source: Doramagic discovery, validation, and Project Pack records
Community Discussion Evidence
These external discussion links are review inputs, not standalone proof that the project is production-ready.
Count of project-level external discussion links exposed on this manual page.
Open the linked issues or discussions before treating the pack as ready for your environment.
Community Discussion Evidence
Doramagic exposes project-level community discussion separately from official documentation. Review these links before using conductor with real data or production workflows.
- [[BUG]: SWITCH task goes through defaultCase if other cases are empty](https://github.com/conductor-oss/conductor/issues/396) - github / github_issue
- v3.31.0-rc.4 - github / github_release
- v3.31.0-rc.3 - github / github_release
- v3.31.0-rc.2 - github / github_release
- v3.30.2 - github / github_release
- v3.31.0-rc.1 - github / github_release
- v3.30.1 - github / github_release
- v3.30.0 - github / github_release
- v3.30.0.rc18 - github / github_release
- v3.30.0.rc17 - github / github_release
- v3.30.0.rc16 - github / github_release
- Capability evidence risk requires verification - GitHub / issue
Source: Project Pack community evidence and pitfall evidence