Doramagic Project Pack · Human Manual

conductor

Conductor is an event driven agentic workflow engine providing durable and highly resilient execution engine for applications and AI Agents

Conductor Overview & System Architecture

Related topics: Workflow Engine, Tasks & Execution Model, Deployment, Configuration & Troubleshooting

Section Related Pages

Continue reading this section for the full explanation and source context.

Section Pluggable persistence

Continue reading this section for the full explanation and source context.

Related topics: Workflow Engine, Tasks & Execution Model, Deployment, Configuration & Troubleshooting

Conductor Overview & System Architecture

1. Purpose and Scope

Conductor is an open-source workflow orchestration engine originally created at Netflix and now maintained as the conductor-oss project under the Apache 2.0 license. The top-level README.md:1-30 positions Conductor as a platform for authoring, executing, and observing durable, stateful workflows that coordinate heterogeneous tasks (HTTP calls, microservices, AI agents, scripts, message-queue work, and more). The repository ships:

  • A server module exposing REST APIs and the orchestration runtime.
  • A common module holding shared data models used by both server and client SDKs.
  • Pluggable persistence modules — one of which is the Postgres-backed implementation in postgres-persistence/.
  • Optional community-contributed integrations such as the Kafka system tasks described in kafka/README.md:1-15.
  • A growing set of AI workflow examples (LLM chat, code agents, research agents) under ai/examples/README.md:1-25 that illustrate how to register and execute LLM-driven workflows through the same REST surface.

A separate project, conductor-oss/rust-sdk, provides an incubating Rust client SDK, reflecting the project's multi-language direction.

2. High-Level Architecture

Conductor follows a layered, interface-driven design. The orchestration engine is decoupled from any specific datastore, and persistence is selected at boot time through Spring conditional configuration.

flowchart TB
    Client["Client / SDK / UI"] -->|REST + JSON| Server["Conductor Server<br/>(Spring Boot)"]
    Server --> Core["Core Orchestration<br/>(WorkflowExecutor, DeciderService,<br/>SystemTask workers)"]
    Core --> DAO["DAO Interfaces<br/>(MetadataDAO, ExecutionDAO,<br/>QueueDAO, IndexDAO)"]
    DAO --> Pg["Postgres DAO<br/>(PostgresMetadataDAO,<br/>PostgresExecutionDAO,<br/>PostgresQueueDAO,<br/>PostgresIndexDAO)"]
    Pg --> Db[("PostgreSQL")]
    Core --> Async["Async Queues<br/>(PostgresQueueDAO listener)"]
    Async --> Pg

The common module defines the public-facing data models — WorkflowDef, TaskDef, Workflow, Task, TaskResult, and friends — and ships JSON Schemas for client-side validation. Source: schemas/README.md:1-30. The recursive WorkflowTask schema enables unlimited nesting of decision, fork-join, do-while, and sub-workflow patterns. The same models are surfaced through Protobuf annotations (e.g. @ProtoMessage / @ProtoField) on classes such as CacheConfig.java:13-22 and ExecutionMetadata.java:19-26, which is why the runtime can support both JSON REST and binary gRPC transports from a single source of truth.

Pluggable persistence

The conductor.db.type Spring property gates which DAO implementation is wired in. For Postgres, see the conditional bean in PostgresConfiguration.java:42-66 (@ConditionalOnProperty(name = "conductor.db.type", havingValue = "postgres")), and the toggleable properties in PostgresProperties.java:18-44 (task-def cache refresh, deadlock retry max, queue-notify toggles, data-migration flag). The README enumerates the supported backend combinations in README.md:1-50: Redis with ES7/ES8/OpenSearch, and Postgres-only or Postgres-plus-ES variants.

3. Persistence Layer (Postgres Implementation)

The Postgres persistence module implements the four core DAO contracts through one shared base class and four concrete DAOs.

  • PostgresBaseDAO.java:36-70 — supplies the shared DataSource, ObjectMapper, and a RetryTemplate used for deadlock/connection retries; it also filters out the DAO's own stack frames from logged exceptions.
  • PostgresMetadataDAO.java:1-40 — stores WorkflowDef, TaskDef, and EventHandler definitions, with an in-memory cache refreshed every taskDefCacheRefreshInterval seconds (default 60s).
  • PostgresExecutionDAO.java:1-40 — persists workflow and task state, plus the rate-limiting and concurrent-execution limit tables; it pushes events through Monitors for metrics.
  • PostgresQueueDAO.java:1-40 — implements task queues; uses a PostgresQueueListener for in-process notifications when experimentalQueueNotify=true and unacks stalled messages after 60s.
  • PostgresIndexDAO.java:1-40 — provides searchable indices of workflows, tasks, and TaskExecLog entries without requiring an external search engine.
Community note: the analogous Elasticsearch 7 path has a long-standing complaint (issue #226) that createIndexesTemplates produced an unbounded number of conductor_task_log_YYYYMMDD indices, eventually exhausting shards. The Postgres IndexDAO sidesteps that by storing the log in relational tables.

4. Core Data Models & Execution Surface

The common module defines the contracts that flow over the wire. Beyond the legacy Workflow / Task shapes, several newer models are worth knowing:

  • WorkflowSummaryExtended.java:18-40 — extends WorkflowSummary so that input and output are returned as Map<String,Object> for JSON consumers, while staying Protobuf-compatible via dedicated field ids.
  • WorkflowRun.java:1-25 and TaskRun.java:1-30 — DTOs used by the signal-handling endpoints to return *what is currently blocking* a workflow or task. Both extend the abstract SignalResponse.java:15-30, which carries workflowId, correlationId, requestId, and the chosen return strategy.
  • WorkflowSignalReturnStrategy.java:8-30 — enum selecting between TARGET_WORKFLOW, BLOCKING_WORKFLOW, BLOCKING_TASK, and BLOCKING_TASK_INPUT. This is the key control for how deep into nested sub-workflows a signal response should be materialised. The accompanying comment notes this was a non-trivial design point once the notification service was introduced.
  • WorkflowTestRequest.java:1-40 — request body for the workflow *test* endpoint; supports per-task mocks (taskRefToMockOutput) and per-sub-workflow mocks (subWorkflowTestRequest), plus simulated executionTime and queueWaitTime for timeout tests.
  • CacheConfig.java:13-40 — per-workflow response-cache configuration (key + ttlInSecond).

5. Build, Runtime, and Operational Gotchas

Per README.md:1-60, building from source requires JDK 21+, Docker Desktop, Node.js 18+, and pnpm (for the UI). Community issue #711 highlights that the bundled docker/server/Dockerfile historically referenced Java 17 in two places, which conflicts with the JavaLanguageVersion.of(21) toolchain declared in build.gradle. Operators upgrading from older Conductor 3.21.x should also be aware of issue #712, in which parent workflows can hang after a sub-workflow completes and require a manual POST /api/workflow/decide/{id} to unblock.

Elasticsearch users should set conductor.indexing.type=elasticsearch explicitly (issue #928) so that the ES7 IndexDAO bean is registered; without it Spring's auto-configuration can fail to start the server. For ES8 (issue #49), use the dedicated config-redis-es8.properties profile.

The latest release line (v3.31.0-rc.4) ships the new ui-next front-end, an updated FAQ entry for duplicate task scheduling on attempt 0, and test-harness fixes — see the release notes for the full change set.

See Also

Source: https://github.com/conductor-oss/conductor / Human Manual

Workflow Engine, Tasks & Execution Model

Related topics: Conductor Overview & System Architecture, AI, LLM, MCP & Vector Database Integration

Section Related Pages

Continue reading this section for the full explanation and source context.

Section Definitions vs. Instances

Continue reading this section for the full explanation and source context.

Section Task Statuses

Continue reading this section for the full explanation and source context.

Related topics: Conductor Overview & System Architecture, AI, LLM, MCP & Vector Database Integration

Workflow Engine, Tasks & Execution Model

Purpose and Scope

Conductor's workflow engine orchestrates long-running, distributed business processes by interpreting declarative workflow definitions and driving tasks through a well-defined lifecycle. A workflow definition (the *blueprint*) is registered once and used to spin off many runtime workflow *instances*. Each instance schedules and tracks the tasks defined inside it, dispatching them to external workers (HTTP, gRPC, polling), system tasks (Kafka, JQ, HTTP, LLM), or built-in control-flow tasks (DECISION, SWITCH, FORK_JOIN, DO_WHILE, SUB_WORKFLOW).

Source: schemas/README.md describes the four core schemas — WorkflowDef, TaskDef, Workflow, and Task — that model this two-layer split between template and runtime.

Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java enumerates every built-in task kind Conductor knows how to interpret, including simple, control-flow, integration, and LLM/MCP tasks.

Core Data Model

Definitions vs. Instances

Schema / ClassRoleNotes
WorkflowDefBlueprint of a workflowRecursive WorkflowTask allows unlimited nesting of DECISION/SWITCH/FORK_JOIN/DO_WHILE/SUB_WORKFLOW Source: schemas/README.md
TaskDefReusable task templateCarries retry count/delay, timeout policy (RETRY/TIME_OUT_WF/ALERT_ONLY), and rate-limit settings Source: schemas/README.md
WorkflowLive runtime instanceCarries status, task list, history (recursive for versioning), priority 0–99, parent/child links Source: schemas/README.md
TaskLive runtime taskCarries status, input/output, workerId, retry counters, iteration, subWorkflowId Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java

Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java shows Task uses @ProtoField annotations for compact binary serialization while remaining compatible with JSON via the proto-aware Jackson module.

Task Statuses

Source: schemas/README.md defines the runtime state machine:

  • In-flight: SCHEDULED, IN_PROGRESS
  • Success: COMPLETED, COMPLETED_WITH_ERRORS, SKIPPED
  • Failure: FAILED, FAILED_WITH_TERMINAL_ERROR, TIMED_OUT, CANCELED

Workflow-level statuses are RUNNING, COMPLETED, FAILED, TIMED_OUT, TERMINATED, PAUSED.

Task Types and Execution

Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java enumerates every supported task kind. The enum is mirrored as string constants (e.g. TASK_TYPE_HTTP = "HTTP") so call sites never hardcode literals. Categories present in the enum include:

CategoryExamples
Simple / WorkerSIMPLE, HTTP, INLINE, LAMBDA
Control-flowDECISION, SWITCH, JOIN, EXCLUSIVE_JOIN, FORK_JOIN, FORK_JOIN_DYNAMIC, DO_WHILE, EVENT, WAIT, HUMAN
Sub-workflowsSUB_WORKFLOW, START_WORKFLOW
AI / LLMLLM_TEXT_COMPLETE, LLM_CHAT_COMPLETE, LLM_INDEX_TEXT, LLM_SEARCH_INDEX, LLM_GENERATE_EMBEDDINGS, LLM_STORE_EMBEDDINGS, LLM_GET_EMBEDDINGS
MCP / agenticLIST_MCP_TOOLS, CALL_MCP_TOOL, PULL_WORKFLOW_MESSAGES

Source: kafka/README.md shows how community-contributed system tasks like KAFKA_PUBLISH and JSON_JQ_TRANSFORM_TASK are modeled — a task declares its type and an inputParameters map. JQ tasks, for example, must include a queryExpression plus any user-named variables.

Source: ai/examples/README.md illustrates end-to-end execution: a workflow such as chat_workflow is registered via POST /api/metadata/workflow, started via POST /api/workflow/{name}, and inspected via GET /api/workflow/{workflowId}. Inputs and outputs are passed as JSON objects, and outputs from earlier tasks are referenced by ${taskRefName.output.field} placeholders.

Execution Flow

flowchart LR
    A[WorkflowDef registered] --> B[POST /api/workflow/name]
    B --> C[Workflow instance created]
    C --> D[Engine maps WorkflowTasks -> Tasks]
    D --> E{Task type}
    E -->|Simple/HTTP| F[Worker polls & completes]
    E -->|System task| G[In-process executor<br/>HTTP, JQ, Kafka, LLM]
    E -->|Control-flow| H[Branch / fork / loop / sub-workflow]
    F & G & H --> I[Task status updated]
    I --> J{Workflow complete?}
    J -->|No| D
    J -->|Yes| K[Workflow terminal status]

Source: schemas/README.md confirms that nested control-flow tasks — decisionCases, defaultCase, forkTasks, loopOver, loopCondition, subWorkflowParam — are encoded directly in the recursive WorkflowTask definition.

Source: common/src/main/java/com/netflix/conductor/common/metadata/events/EventHandler.java shows how events plug into execution: an EventHandler carries an actions list and a nested TaskDetails message that references the producing workflowId, taskRefName, taskId, and output, enabling event-driven task completion without polling.

Sub-Workflow and Async Patterns

Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java reserves fields for sub-workflow coordination: subWorkflowId and workflowInstanceId link a SUB_WORKFLOW task to the child workflow it spawned, while workflowPriority lets high-priority runs preempt lower ones.

A common community-reported failure mode (referenced in issue #712 *"The workflow hangs after running Sub workflow"*) is that the parent workflow does not automatically resume after the child completes until an external POST /api/workflow/decide/{workflowId} is issued. This is consistent with the explicit linking fields in Task and emphasizes that sub-workflow completion must be observed through the normal async-decision loop.

Serialization and Wire Format

Source: common/src/main/java/com/netflix/conductor/common/jackson/JsonProtoModule.java explains how Any protobufs (used for inputMessage and outputMessage on Task, and outputMessage on EventHandler.TaskDetails) are exposed over REST without requiring the server to know every embedded type. The module encodes an Any as a tuple of type-URL plus raw base64 payload so it can be transparently relayed to clients.

Source: annotations-processor/README.md notes that the protobuf schema itself is generated from @ProtoMessage/@ProtoField annotations, keeping the binary and JSON representations in lockstep.

Configuration and Persistence Touchpoints

Although the engine itself is storage-agnostic, the runtime data (Workflow, Task) lands in a configurable indexing backend. Source: es8-persistence/README.md describes the ES8 module that stores ${indexPrefix}_workflow, ${indexPrefix}_task, ${indexPrefix}_task_log, ${indexPrefix}_message, and ${indexPrefix}_event indices via write aliases and an ILM policy. Community issues #226 and #928 highlight two operational pitfalls: ES7 persistence creating unbounded time-based indices when rollover is misconfigured, and ES7 Docker images that omit conductor.indexing.type=elasticsearch so Spring fails to wire the IndexDAO bean — both underscore the importance of matching indexing type to the selected persistence module.

Source: README.md lists the supported backend combinations in docker/server/config/, including Redis+ES7/ES8/OpenSearch, Postgres, Postgres+ES7, and MySQL+ES7.

See Also

Source: https://github.com/conductor-oss/conductor / Human Manual

AI, LLM, MCP & Vector Database Integration

Related topics: Workflow Engine, Tasks & Execution Model

Section Related Pages

Continue reading this section for the full explanation and source context.

Section Model Context Protocol (MCP)

Continue reading this section for the full explanation and source context.

Section Vector Database Operations

Continue reading this section for the full explanation and source context.

Section Schema and Input Validation

Continue reading this section for the full explanation and source context.

Related topics: Workflow Engine, Tasks & Execution Model

AI, LLM, MCP & Vector Database Integration

Overview

Conductor exposes AI capabilities as first-class system task types so that language models, embedding stores, and external tool servers can be orchestrated the same way as any HTTP or business task. Workflow authors declare prompts, model parameters, vector-store identifiers, and MCP server URLs in JSON; the Conductor runtime handles scheduling, retry, persistence, and observability for each step. Source: ai/examples/README.md:1-30

This design is part of the project's "ship agents, not framework code" positioning: workers remain plain code in any language, while LLM-driven steps are declarative and machine-readable, which lets LLMs themselves generate and compose workflows. Source: README.md:35-50

LLM, Embedding, and MCP Task Types

The catalog of AI-related task types is defined as enum constants and string constants in the shared metadata module:

Task TypeConstant StringPurpose
Text completionLLM_TEXT_COMPLETESingle-prompt text generation
Chat completionLLM_CHAT_COMPLETEMulti-turn chat with system / user / assistant messages
Index textLLM_INDEX_TEXTPush document chunks into a vector index
Search indexLLM_SEARCH_INDEXSimilarity / RAG search over a vector index
Generate embeddingsLLM_GENERATE_EMBEDDINGSProduce vector embeddings from text
Store embeddingsLLM_STORE_EMBEDDINGSPersist pre-computed embeddings
Get embeddingsLLM_GET_EMBEDDINGSRetrieve stored embeddings by reference
List MCP toolsLIST_MCP_TOOLSDiscover tools exposed by an MCP server
Call MCP toolCALL_MCP_TOOLInvoke a discovered tool on an MCP server
Pull workflow messagesPULL_WORKFLOW_MESSAGESDrain queued chat messages for a workflow

Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/TaskType.java:1-30

Because these are regular Task instances, the same runtime semantics apply: scheduled/started/updated timestamps, retry policies, queue-wait accounting, and worker attribution. Source: common/src/main/java/com/netflix/conductor/common/metadata/tasks/Task.java:30-50

Provider and Pattern Examples

The ai/examples/ directory ships 22 reference workflows demonstrating the supported providers and orchestration patterns:

  • Chat and reasoning01-chat-completion.json (OpenAI, Anthropic, Gemini, Vertex AI), 20-extended-thinking.json (Anthropic with a thinking-token budget), 22-multi-turn-chain.json (OpenAI previousResponseId chaining).
  • Embeddings and RAG02-embeddings.json, 07-rag-complete.json (OpenAI embeddings with PostgreSQL as the vector store).
  • Tool and agent patterns08-mcp-list-tools.json, 09-mcp-call-tool.json (weather), 10-mcp-ai-agent.json (LLM + MCP).
  • Media generation11-video-openai-sora.json (async), 12-video-gemini-veo.json (async), 13-image-to-video-pipeline.json, 14-stabilityai-image.json (SD3.5).
  • Document and research15-pdf-generation.json, 16-llm-to-pdf-pipeline.json, 21-web-search-research-agent.json (web search → synthesize → PDF).
  • Specialized17-web-search.json, 18-code-execution.json (Gemini sandbox), 19-coding-agent.json (plan → write & run code → review).

Source: ai/examples/README.md:1-18

The README's flagship pattern is an autonomous "think-act" agent: a LIST_MCP_TOOLS task discovers available tools, a DO_WHILE task alternates a LLM_CHAT_COMPLETE ("think") step with a tool invocation, and the loop exits when the model emits done: true. The MCP server URL is passed in as ${workflow.input.mcpServerUrl} and the discovered tool list is injected into the system prompt as ${discover.output.tools}. Source: README.md:40-65

MCP, Vector Stores, and Schemas

Model Context Protocol (MCP)

MCP integration is modeled as two task types — LIST_MCP_TOOLS and CALL_MCP_TOOL — so an external MCP server is just another declarative step in the workflow. This keeps the agent loop in the same JSON definition file as every other task, rather than requiring a custom worker. Source: README.md:40-65

Vector Database Operations

Vector storage and retrieval are decomposed into three operations: LLM_GENERATE_EMBEDDINGS produces vectors, LLM_INDEX_TEXT writes them, and LLM_SEARCH_INDEX queries them. Example 07-rag-complete.json wires all three together with OpenAI embeddings and PostgreSQL as the backing store. Source: ai/examples/README.md:5-12

Schema and Input Validation

LLM task payloads are JSON objects (for example messages: [...], tools: [...]) and can be guarded by a SchemaDef that supports inline JSON, AVRO, Protobuf, or an external registry reference via externalRef. Source: common/src/main/java/com/netflix/conductor/common/metadata/SchemaDef.java:1-15 The repository additionally publishes JSON Schemas (Draft 07) for the core models so editors can validate workflow and task definitions before registration. Source: schemas/README.md:1-20

Using the AI Tasks

A typical lifecycle is identical to any Conductor workflow — register the definition, start an execution, then poll for the result:

# 1. Register the workflow definition
curl -X POST 'http://localhost:8080/api/metadata/workflow' \
  -H 'Content-Type: application/json' \
  -d @01-chat-completion.json

# 2. Start an execution
curl -X POST 'http://localhost:8080/api/workflow/chat_workflow' \
  -H 'Content-Type: application/json' \
  -d '{}'

# 3. Check execution status
curl -X GET 'http://localhost:8080/api/workflow/{workflowId}'

Source: ai/examples/README.md:18-30

The new ui-next console (React + Vite, served at http://localhost:1234 in dev mode) provides the front-end for inspecting these workflows, including executions that mix LLM, MCP, and vector-store tasks. Source: ui-next/README.md:1-20

Common Failure Modes

  • Async media generation11-video-openai-sora.json and 12-video-gemini-veo.json are explicitly asynchronous; downstream tasks must poll for completion rather than expecting a synchronous result. Source: ai/examples/README.md:8-10
  • Provider configuration — API keys, model identifiers, and (for Vertex AI) region are supplied at runtime; missing credentials surface as task failures rather than startup errors.
  • MCP server reachabilityLIST_MCP_TOOLS is the entry point of the agent loop, so an unreachable MCP server fails the discover step and halts the whole workflow, which is also why it doubles as a connectivity check.
  • Schema drift — Providers evolve their message and tool-calling schemas; updates should be reflected in the corresponding JSON Schema files before re-registering definitions. Source: schemas/README.md:1-20

See Also

Source: https://github.com/conductor-oss/conductor / Human Manual

Deployment, Configuration & Troubleshooting

Related topics: Conductor Overview & System Architecture, Workflow Engine, Tasks & Execution Model

Section Related Pages

Continue reading this section for the full explanation and source context.

Section Backend Configurations

Continue reading this section for the full explanation and source context.

Section Indexing Backend Selection

Continue reading this section for the full explanation and source context.

Section Build and Runtime Requirements

Continue reading this section for the full explanation and source context.

Related topics: Conductor Overview & System Architecture, Workflow Engine, Tasks & Execution Model

Deployment, Configuration & Troubleshooting

Overview

Conductor OSS is a workflow orchestration engine that ships as a multi-module Gradle project. The server (a Spring Boot application) is paired with pluggable persistence backends: a primary datastore (Redis, Postgres, MySQL, or Cassandra) for queue and runtime state, and an indexing backend (Elasticsearch 7, Elasticsearch 8, or OpenSearch) for search and visibility. The current release line is v3.31.0-rc.4 as listed in README.md. Deployment, configuration, and troubleshooting are tightly coupled: a successful installation depends on selecting the correct backend module, providing the right indexing type property, and matching the JDK version expected by the build.

Deployment Topology

The default Docker Compose stack brings up the conductor server, a database, an index store, and the new ui-next console, all wired through Spring Boot auto-configuration.

flowchart LR
    Worker[External Workers] -->|HTTP poll / task ack| Server[Conductor Server]
    UI[ui-next React UI] -->|REST| Server
    CLI[Conductor CLI / curl] -->|REST| Server
    Server -->|queue + state| DB[(Redis / Postgres / MySQL / Cassandra)]
    Server -->|index workflow + task| IDX[(Elasticsearch 7 / 8 / OpenSearch)]
    Server -.->|annotation processor| AP[protobuf generation]

The ui-next workspace replaces the legacy ui in published releases; both are still present in the repository, but recent publish pipelines now point at ui-next (see the v3.31.0-rc.4 changelog referencing PR #1168 in README.md).

Backend Configurations

Six pre-bundled property files live in docker/server/config/ and are selected at container startup. The choice of file determines both the primary datastore and the indexing backend, and the indexing type property must match the file in use.

Backend combinationProperty file
Redis + Elasticsearch 7 (default)config-redis.properties
Redis + Elasticsearch 8config-redis-es8.properties
Redis + OpenSearchconfig-redis-os.properties
Postgres (no ES)config-postgres.properties
Postgres + Elasticsearch 7config-postgres-es7.properties
MySQL + Elasticsearch 7config-mysql.properties

Source: README.md (Backend Configuration table).

Configuration

Indexing Backend Selection

The conductor.indexing.type property is the single switch that activates the correct IndexDAO bean. Setting it to elasticsearch activates the ES7 module, elasticsearch8 activates the ES8 module, and opensearch activates the OpenSearch integration. The ES8 module additionally uses composable index templates, write aliases, and an ILM policy (max_primary_shard_size=50gb) backed by the elasticsearch-java 8.19.x client. Source: es8-persistence/README.md.

When building from source the matching Gradle module must be selected to avoid Lucene conflicts:

./gradlew :conductor-server:bootJar -PindexingBackend=elasticsearch8

-PindexingBackend=es8 is accepted as a shortcut. Source: es8-persistence/README.md.

Build and Runtime Requirements

Build from source requires Docker Desktop, JDK 21+, and Node.js 18+ with pnpm for the UI. Source: README.md (Build From Source section). The legacy ui package declares node >=14.17.0 while ui-next uses a peerDependencies list aligned with React 18 and MUI v7. Sources: ui/package.json and ui-next/package.json.

Annotation Processing

Custom task and workflow definitions compiled against the project go through the annotations-processor module, which generates protobuf descriptors from @ProtoMessage/@ProtoField annotations. The generated wire format is what enables cross-language gRPC clients to interoperate with the server. Source: annotations-processor/README.md.

Schema Validation

JSON Schemas under schemas/ mirror the Java model classes and can be used by IDEs (VS Code, IntelliJ) for editor-time validation of workflow and task definitions. They conform to JSON Schema Draft 07 and support $ref for composition, mirroring the recursive WorkflowTask structure. Source: schemas/README.md.

Common Issues and Troubleshooting

1. Server fails to start: "no IndexDAO bean"

The most common startup failure, tracked in issue #928, is an absent conductor.indexing.type=elasticsearch line in the ES7 Docker config. Without it, Spring does not instantiate the ES7 IndexDAO bean and bootstrap aborts. Fix: explicitly set the indexing type in the active properties file to match the bundled config (e.g. elasticsearch, elasticsearch8, or opensearch). Source: README.md (Backend Configuration).

2. JDK mismatch in Docker image

Issue #711 reports that docker/server/Dockerfile references Java 17 while build.gradle requires Java 21 (sourceCompatibility = JavaVersion.VERSION_21). The runtime image must be rebuilt against a JDK 21 base image to match the toolchain; otherwise the boot jar will not execute.

3. Parent workflow hangs after a sub-workflow completes

Issue #712 describes a regression in v3.21.23 Docker images where the parent workflow stops progressing after the child SUB_WORKFLOW task completes. The parent can be unstuck by issuing POST /api/workflow/decide/{workflowId}. Sub-workflow references are tracked via the subWorkflowParam field and the SUB_WORKFLOW task type constant declared in TaskType.java.

4. Unbounded index growth in ES7 persistence

Issue #226 documents an older ElasticSearchRestDAOV7 path that creates a new conductor_task_log_YYYYMMDD index daily without rollover, leading to thousands of shards. The modern ES8 module fixes this by using composable index templates plus an ILM policy capped at 50gb primary shard size. Recommendation: migrate to the ES8 backend, or configure a manual rollover/ILM policy on the ES7 indices. Source: es8-persistence/README.md.

5. ES8 compatibility

Long-running request #49 asking for ES8 support is now resolved via the dedicated es8-persistence module, which uses the official elasticsearch-java 8.19.11 client and ships with a pinned 8.19.x Docker image. Source: es8-persistence/README.md.

6. Duplicate task scheduling (attempt 0 × 2)

A new FAQ entry (added in v3.31.0-rc.4 per PR #1173) addresses a duplicate scheduling symptom where the same task appears twice at attempt 0. The typical cause is a worker acking the task before the queue listener has finished writing the SCHEDULED record; polling workers should treat the first GET /tasks response as authoritative and use idempotency keys on the ack call.

Operational Tips

See Also

  • Architecture overview
  • Task types reference
  • Workflow definition schema
  • Elasticsearch backend modules (ES7, ES8, OpenSearch)

Source: https://github.com/conductor-oss/conductor / Human Manual

Doramagic Pitfall Log

Source-linked risks stay visible on the manual page so the preview does not read like a recommendation.

medium Capability evidence risk requires verification

May increase setup, validation, or first-run risk for the user.

medium Maintenance risk requires verification

May increase setup, validation, or first-run risk for the user.

medium Maintenance risk requires verification

May increase setup, validation, or first-run risk for the user.

medium Security or permission risk requires verification

May increase setup, validation, or first-run risk for the user.

Doramagic Pitfall Log

Found 7 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification.

1. Capability evidence risk: Capability evidence risk requires verification

  • Severity: medium
  • Finding: README/documentation is current enough for a first validation pass.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: capability.assumptions | https://github.com/conductor-oss/conductor

2. Maintenance risk: Maintenance risk requires verification

  • Severity: medium
  • Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: community_evidence:github | https://github.com/conductor-oss/conductor/issues/396

3. Maintenance risk: Maintenance risk requires verification

  • Severity: medium
  • Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor

4. Security or permission risk: Security or permission risk requires verification

  • Severity: medium
  • Finding: no_demo
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: downstream_validation.risk_items | https://github.com/conductor-oss/conductor

5. Security or permission risk: Security or permission risk requires verification

  • Severity: medium
  • Finding: no_demo
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: risks.scoring_risks | https://github.com/conductor-oss/conductor

6. Maintenance risk: Maintenance risk requires verification

  • Severity: low
  • Finding: issue_or_pr_quality=unknown。
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor

7. Maintenance risk: Maintenance risk requires verification

  • Severity: low
  • Finding: release_recency=unknown。
  • User impact: May increase setup, validation, or first-run risk for the user.
  • Recommended check: Reproduce the official install and quickstart path in an isolated environment.
  • Evidence: evidence.maintainer_signals | https://github.com/conductor-oss/conductor

Source: Doramagic discovery, validation, and Project Pack records

Community Discussion Evidence

These external discussion links are review inputs, not standalone proof that the project is production-ready.

Sources 12

Count of project-level external discussion links exposed on this manual page.

Use Review before install

Open the linked issues or discussions before treating the pack as ready for your environment.

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using conductor with real data or production workflows.

Source: Project Pack community evidence and pitfall evidence