# https://github.com/PrefectHQ/prefect Project Manual

Generated at: 2026-06-20 19:35:08 UTC

## Table of Contents

- [Core Concepts and SDK: Flows, Tasks, States, and Results](#page-1)
- [System Architecture: Server, API, Database, and Events Pipeline](#page-2)
- [Deployment, Workers, and Infrastructure](#page-3)
- [Concurrency, Events, and Common Failure Modes](#page-4)

<a id='page-1'></a>

## Core Concepts and SDK: Flows, Tasks, States, and Results

### Related Pages

Related topics: [System Architecture: Server, API, Database, and Events Pipeline](#page-2), [Deployment, Workers, and Infrastructure](#page-3)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/prefect/flows.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flows.py)
- [src/prefect/tasks.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/tasks.py)
- [src/prefect/flow_engine.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py)
- [src/prefect/task_engine.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py)
- [src/prefect/states.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/states.py)
- [src/prefect/results.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/results.py)
- [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md)
- [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py)
- [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py)
- [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py)
- [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py)
- [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py)
- [src/prefect/server/api/block_documents.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_documents.py)
- [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py)
</details>

# Core Concepts and SDK: Flows, Tasks, States, and Results

## Overview

Prefect is a workflow orchestration framework that elevates Python scripts into production-grade data pipelines, exposing features such as scheduling, caching, retries, and event-based automations. Source: [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md). Four primitives form the conceptual spine of every Prefect program: **Flows**, **Tasks**, **States**, and **Results**. Together they describe *what runs*, *what happens while it runs*, *how progress is observed*, and *how outputs are persisted*.

The server surface mirrors these primitives: routers such as `flows`, `flow_runs`, `flow_run_states`, `task_runs`, `task_run_states`, `task_workers`, and `deployments` are mounted under a single FastAPI application in `src/prefect/server/api/server.py`. Source: [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py).

## Flows, Tasks, and the Execution Engines

### Flows

A **flow** is the top-level unit of work — a Python function decorated (or constructed) to be tracked. The SDK in `src/prefect/flows.py` exposes the `Flow` class and `@flow` decorator; the engine in `src/prefect/flow_engine.py` is responsible for running it, submitting child tasks, and reporting state transitions to the API.

### Tasks

A **task** is a discrete, cacheable unit of work invoked from within a flow. Tasks are defined in `src/prefect/tasks.py` and executed by `src/prefect/task_engine.py`. Tasks support retries, caching, and concurrent submission via `.submit()`. The `task_workers` router exposes server-side worker coordination for tasks. Source: [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py).

### Engine responsibilities

The flow engine and task engine share an analogous structure: they translate a Python call into a `FlowRun` or `TaskRun` record, drive it through the state machine, and persist the result. State transitions for both run types are surfaced to the server via the `flow_run_states` and `task_run_states` routers. Source: [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py).

```mermaid
flowchart LR
    A[User Code] -->|@flow decorator| B(Flow Object)
    A -->|@task decorator| C(Task Object)
    B --> D[flow_engine]
    C --> E[task_engine]
    D --> F[(FlowRun + FlowRunState)]
    E --> G[(TaskRun + TaskRunState)]
    F --> H[Result persistence]
    G --> H
    H --> I[(Block / Result store)]
```

## States and Run Lifecycle

### The state machine

A **state** describes the current condition of a flow or task run (`Pending`, `Running`, `Completed`, `Failed`, `Cancelled`, `Cancelling`, `Crashed`, etc.). States are modeled in `src/prefect/states.py` and persisted through the `flow_run_states` / `task_run_states` endpoints.

The 3.7.5 release explicitly added a "cancelling timeout cleanup producer," highlighting that `CANCELLING` is a real, transitional state — not just an alias for `CANCELLED`. Operators must distinguish between runs that have begun cancellation work and those that have finished it. Source: [Release 3.7.5](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5).

### Run history queries

The server exposes a unified `run_history` endpoint that aggregates runs by interval. It accepts filters for `flows`, `flow_runs`, `task_runs`, `deployments`, `work_pools`, and `work_queues`, and enforces a minimum `history_interval` of one second (SQLite cannot reliably bucket sub-second intervals). Source: [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py).

### SQLite lock handling

Because Prefect can run against SQLite for local development, the server installs a filter that downgrades noisy `SQLITE_BUSY` errors to a configurable log level. Source: [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py).

## Results, Persistence, and Configuration

### Result persistence

A **result** is the value returned by a task or flow run. The SDK abstraction lives in `src/prefect/results.py`, which chooses a result storage backend and serializes payloads. Block documents — the reusable, typed configuration objects referenced by many result stores — are managed by the `block_documents` and `block_schemas` routers. Source: [src/prefect/server/api/block_documents.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_documents.py); [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py).

### Job variables and deployment validation

When a flow runs on a worker, the server validates job variables contributed from three layers: the work pool's base job template, the deployment, and the flow run — applied in that order, with flow run values taking precedence. Source: [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py). The `deployments` router coordinates this validation, including JSON-schema hydration for block references. Source: [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py).

### Known failure modes

Community reports identify two recurring pitfalls related to these primitives:

- **Concurrent subflow result collisions** — When subflows are invoked concurrently (the common `task.submit()` wrapper pattern), the virtual task that tracks each subflow can adopt another subflow's persisted result because `dynamic_key` derivation is positional. Source: [Issue #22259](https://github.com/PrefectHQ/prefect/issues/22259).
- **Test harness breakage** — The `PrefectRouter` test helper previously lacked a `routes` attribute expected by user code, surfacing as an `AttributeError` during fixture setup. Source: [Issue #22307](https://github.com/PrefectHQ/prefect/issues/22307).

## See Also

- [Server API: Deployments and Work Pools](https://docs.prefect.io/v3/concepts/deployments)
- [States and State Transitions](https://docs.prefect.io/v3/concepts/states)
- [Tasks, Caching, and Concurrency](https://docs.prefect.io/v3/concepts/tasks)
- [Results and Result Storage](https://docs.prefect.io/v3/concepts/results)
- [Release notes for 3.7.x](https://github.com/PrefectHQ/prefect/releases)

---

<a id='page-2'></a>

## System Architecture: Server, API, Database, and Events Pipeline

### Related Pages

Related topics: [Core Concepts and SDK: Flows, Tasks, States, and Results](#page-1), [Concurrency, Events, and Common Failure Modes](#page-4)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md)
- [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py)
- [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py)
- [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py)
- [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py)
- [src/prefect/server/api/block_documents.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_documents.py)
- [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py)
- [src/prefect/server/api/artifacts.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/artifacts.py)
- [src/prefect/server/api/task_run_states.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/task_run_states.py)
- [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py)
- [src/prefect/server/api/clients.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/clients.py)
- [src/prefect/server/api/ui/flow_runs.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/ui/flow_runs.py)
- [src/prefect/runtime/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/runtime/__init__.py)
</details>

# System Architecture: Server, API, Database, and Events Pipeline

## Overview and Component Layers

Prefect is a workflow orchestration framework that exposes a FastAPI-based HTTP server for managing flows, task runs, deployments, work pools, blocks, automations, variables, concurrency limits, and artifacts. The system is organized into four cooperating layers that work end-to-end whenever a client invokes an orchestration action. Source: [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md) describes the project as a framework that "elevate[s] a script into a production workflow" with scheduling, caching, retries, and event-based automations.

```mermaid
flowchart LR
    A[Python Client / CLI] -->|HTTP| B[FastAPI Server]
    B --> C[API Routers]
    C --> D[ORM Models / Database]
    D --> E[(SQLite / Postgres)]
    B --> F[Messaging Broker]
    F --> G[Events Stream]
    G --> H[Automations / WebSocket Subscribers]
```

The four layers are:

1. **HTTP / API layer** — FastAPI routers grouped by domain (`flow_runs`, `task_runs`, `deployments`, `events`, etc.) that translate REST calls into ORM operations.
2. **Persistence layer** — SQLAlchemy ORM models backed by either SQLite (development/test) or Postgres (production), accessed through `PrefectDBInterface`.
3. **Messaging layer** — A pluggable broker (in-memory or Redis via `prefect_redis.messaging`) used by the events pipeline for ephemeral subscriptions and durable event delivery.
4. **Runtime layer** — Modules under `prefect.runtime` that expose dynamic attributes for the current flow run, task run, and deployment to user code. Source: [src/prefect/runtime/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/runtime/__init__.py) re-exports `prefect.runtime.deployment`, `prefect.runtime.flow_run`, and `prefect.runtime.task_run` so a flow can introspect its execution context.

## FastAPI Server Composition

The `create_app` factory in [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py) wires every domain router onto a single FastAPI application. The import block at the top of that file is the canonical list of mounted sub-applications:

| Router | Module | Responsibility |
| --- | --- | --- |
| `admin` | `api/admin` | Maintenance endpoints |
| `artifacts` | `api/artifacts` | Artifact CRUD (see [src/prefect/server/api/artifacts.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/artifacts.py)) |
| `automations` | `api/automations` | Triggered actions and event rules |
| `block_capabilities`, `block_documents`, `block_schemas`, `block_types` | `api/blocks/*` | Block storage (see [src/prefect/server/api/block_documents.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_documents.py), [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py)) |
| `concurrency_limits`, `concurrency_limits_v2` | `api/*` | Resource throttling |
| `deployments` | `api/deployments` | Deployment lifecycle (see [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py)) |
| `events` | `api/events` | Event ingest + replay for the messaging pipeline |
| `flow_runs`, `task_runs`, `flow_run_states`, `task_run_states` | `api/*` | Orchestration primitives (see [src/prefect/server/api/task_run_states.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/task_run_states.py)) |
| `task_workers` | `api/task_workers` | In-process task submission workers |
| `variables`, `work_queues`, `workers` | `api/*` | Configuration and worker pool management |
| `ui.*` | `api/ui/*` | Dashboard-friendly projections (see [src/prefect/server/api/ui/flow_runs.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/ui/flow_runs.py)) |

Every router is built on the custom `PrefectRouter` wrapper, which the project uses to standardize route registration. The complete enumeration of expected routers lives in [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py) (used for both `TYPE_CHECKING` and lazy `importlib` import via `__getattr__`).

The server also installs a SQLite-specific logging filter — `_SQLiteLockedOperationalErrorFilter` in [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py) — that suppresses `SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT` errors when `PREFECT_SERVER_LOG_RETRYABLE_ERRORS=False`, preventing transient lock contention from filling logs.

## Database and ORM Layer

All routers funnel persistence through `PrefectDBInterface`, which is dependency-injected via FastAPI `Depends(provide_database_interface)`. For example, [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py) opens a transactional session with `async with db.session_context(begin_transaction=True) as session`, then delegates to `models.deployments` to read or mutate rows. The same pattern appears in `artifacts.py`, `block_documents.py`, and `task_run_states.py`.

Cross-cutting concerns live alongside the routers:

- **Job-variable validation** — [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py) centralizes the order in which overrides apply (work pool → deployment → flow run) and documents why required-field checks are deferred until the terminal flow-run layer.
- **Run history aggregation** — [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py) provides time-bucketed aggregations for dashboards, and enforces a minimum interval of 1 second because "SQLite has issues with very small intervals (by 0.001 seconds it stops incrementing the interval)".
- **Internal clients** — [src/prefect/server/api/clients.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/clients.py) wraps `PrefectHttpxAsyncClient` for server-side sub-requests (e.g., the worker heartbeat flow), pre-resolving the auth string from server settings so internal calls are authorized consistently.

Release 3.7.5 highlighted several database-layer hardening fixes that are directly relevant to this architecture: an index on `event_resources(occurred)` for vacuum performance (PR #22250), FastAPI 0.137 compatibility that skips route deletion on the server router (3.7.5.dev4), and a "cancelling timeout cleanup producer" enhancement (PR #22238) that prevents flow runs from being stranded in `CANCELLING` after a timeout. Source: [3.7.5 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5).

## Events Pipeline and Operational Concerns

The `events` router feeds a broker-backed stream that downstream consumers read from — including automations, the UI, and any client calling `ephemeral_subscription()` for WebSocket updates. When `PREFECT_MESSAGING_BROKER=prefect_redis.messaging`, ephemeral subscriptions create Redis consumer groups named `ephemeral-{hostname}...` on the `events` stream. Source: [issue #22136](https://github.com/PrefectHQ/prefect/issues/22136) describes how ungraceful shutdowns can leak those groups, pinning the stream and preventing trimming because Redis Streams only trim past the last-acknowledged offset for every consumer group.

A second community-reported class of bugs surfaces at the orchestration layer above the database:

- **Subflow dynamic_key collisions** — [issue #22259](https://github.com/PrefectHQ/prefect/issues/22259) shows that concurrent subflow invocations can adopt the wrong subflow's persisted result on parent retry because the virtual task run's `dynamic_key` is derived positionally. The fix requires per-flow-run, call-site-keyed identifiers rather than positional counters.
- **Test harness breakage** — [issue #22307](https://github.com/PrefectHQ/prefect/issues/22307) reports that `PrefectRouter` did not expose the `routes` attribute expected by Starlette test utilities, raising `AttributeError: 'PrefectRouter' object has no attribute 'routes'` when running against a temporary database.
- **Lifecycle event coverage** — [3.7.4 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.4) added `created`, `updated`, and `deleted` events for variables, flows, block documents, block types, concurrency limits, artifact collections, and automations (PR #22226), giving the events pipeline end-to-end visibility into domain-object mutation.

For day-to-day operations, the practical takeaways are: prefer Postgres in production so that stream trimming and concurrency primitives behave predictably; monitor ephemeral Redis consumer groups during rolling deploys; and rely on the per-router dependency-injected `db: PrefectDBInterface` rather than constructing sessions directly, because that path is what every model function in `prefect.server.models` is designed against.

## See Also

- [3.7.4 — From cradle to grave release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.4)
- [3.7.5 — No run left CANCELLING release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5)
- [Prefect documentation: Blocks](https://docs.prefect.io/v3/concepts/blocks)
- [Prefect documentation: Deployments](https://docs.prefect.io/v3/concepts/deployments)
- [Prefect documentation: Self-host the server](https://docs.prefect.io/latest/manage/self-host/)

---

<a id='page-3'></a>

## Deployment, Workers, and Infrastructure

### Related Pages

Related topics: [Core Concepts and SDK: Flows, Tasks, States, and Results](#page-1), [System Architecture: Server, API, Database, and Events Pipeline](#page-2)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py)
- [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py)
- [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py)
- [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py)
- [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py)
- [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py)
- [src/prefect/runtime/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/runtime/__init__.py)
- [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md)
- [src/integrations/prefect-kubernetes/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/README.md)
- [src/integrations/prefect-aws/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-aws/README.md)
- [src/integrations/prefect-slack/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-slack/README.md)
- [src/integrations/prefect-github/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-github/README.md)
- [src/integrations/prefect-gitlab/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-gitlab/README.md)
- [src/integrations/prefect-bitbucket/README.md](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-bitbucket/README.md)
</details>

# Deployment, Workers, and Infrastructure

## Overview

Prefect is a workflow orchestration framework that turns Python scripts into resilient, observable production workflows. The **Deployment, Workers, and Infrastructure** subsystem is the bridge between authored `@flow` code and the execution environment that actually runs it. A *deployment* is the stored, server-side description of a flow (entrypoint, parameters, schedule, tags, job variables). A *worker* is a long-running process that polls a work pool for scheduled or pending flow runs, materializes the execution *infrastructure* (a container, a Kubernetes job, a process), and submits the run. Together they implement the "from cradle to grave" lifecycle that the 3.7.4 release notes explicitly call out ([3.7.4 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.4)).

The server-side surface for this subsystem is defined under `src/prefect/server/api/`. The package's `__init__.py` enumerates the public API routers; among them, `deployments`, `workers`, `work_queues`, `flow_runs`, `task_workers`, and `task_runs` are the ones that back the deployment/worker domain. Source: [src/prefect/server/api/__init__.py:7-48](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py).

## Server API for Deployments

The `deployments` router is mounted at prefix `/deployments` and exposes CRUD, scheduling, and run-creation endpoints. It is wired into the FastAPI app alongside workers, work queues, and flow run routers in the server composition. Source: [src/prefect/server/api/deployments.py:39-43](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py), [src/prefect/server/api/server.py:1-19](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py).

When a deployment is created or updated, the server validates the supplied `job_variables` against the work pool's base job template. The order of precedence is intentional and well-documented in the validation module: **work pool defaults → deployment overrides → flow run overrides**, with flow runs being the only point at which `required` schema fields are enforced. Source: [src/prefect/server/api/validation.py:1-30](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py).

```python
# Validation entry points (illustrative names)
validate_job_variables_for_deployment(...)        # ignores required fields
validate_job_variables_for_deployment_flow_run(...)  # enforces required fields
validate_job_variables_for_run_deployment_action(...)  # enforces required fields
```

The `run_history` endpoint is also part of the deployment observability story: it returns aggregated state histograms for flow and task runs over a time interval, which workers and schedulers can use to reason about throughput. Source: [src/prefect/server/api/run_history.py:35-58](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py).

## Workers, Work Pools, and Infrastructure Providers

The worker abstractions themselves are exercised through the `workers` and `work_queues` routers mounted in the same composition. Workers in Prefect are pluggable: the core `prefect` package ships with `Process` and `Docker` worker types, while integration packages ship typed infrastructure blocks for cloud and orchestrator runtimes.

| Integration | Infrastructure surface |
|---|---|
| `prefect-kubernetes` | Kubernetes Jobs, Helm-managed deployments ([README](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-kubernetes/README.md)) |
| `prefect-aws` | ECS, ECS Fargate, and step functions (see [README](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-aws/README.md)) |
| `prefect-slack` | Notifications, not execution infrastructure ([README](https://github.com/PrefectHQ/prefect/blob/main/src/integrations/prefect-slack/README.md)) |
| `prefect-github` / `prefect-gitlab` / `prefect-bitbucket` | Storage blocks for source checkout |

The server API for worker registration is exposed through `api.workers.router` and `api.work_queues.router`, both of which are imported and mounted in the central `include_router` call. Source: [src/prefect/server/api/server.py:1-19](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py). Block schemas — the typed descriptors that workers hydrate at run time — are managed by `api.block_schemas.router` and are stored with a checksum so identical schemas are deduplicated. Source: [src/prefect/server/api/block_schemas.py:21-58](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py).

### Flow from deployment to run

```mermaid
flowchart LR
    A["@flow function"] --> B["Deployment<br/>(schedule, params, job vars)"]
    B --> C["Work Pool<br/>(base job template)"]
    C --> D["Worker polls<br/>work queue"]
    D --> E["Infrastructure<br/>(Process / Docker / K8s / ECS)"]
    E --> F["Flow Run<br/>(states + result)"]
    F --> G["Observability<br/>(run_history, events)"]
```

## Runtime Context and the User-Facing API

Inside a running flow, deployment-time metadata is exposed through `prefect.runtime`. The package re-exports three submodules (`deployment`, `flow_run`, `task_run`) that let flow code read its own deployment id, parameters, or run identity without explicit plumbing. Source: [src/prefect/runtime/__init__.py:1-15](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/runtime/__init__.py).

```python
from prefect.runtime import deployment
print(f"Running from deployment {deployment.id} with parameters {deployment.parameters}")
```

## Known Issues and Community Pain Points

Several recently merged changes target the worker/infrastructure path directly:

- **Cancelling timeout cleanup** — 3.7.5 added a "cancelling timeout cleanup producer" so runs stuck in `CANCELLING` are reaped by the worker channel rather than leaking ([3.7.5 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5)). The 3.7.4.dev4 release switched the cleanup service to `automatic=True` for the same reason.
- **Worker observability** — 3.7.5 also added backend observability for the worker channel and cleanup queue, helping operators distinguish "no scheduled runs" from "worker is wedged."
- **FastAPI 0.137 compatibility** — 3.7.5.dev4 skipped stale route deletions in the deployment router to remain compatible with newer FastAPI; deployments that hit a server under FastAPI 0.137 should upgrade ([3.7.5.dev4 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev4)).
- **Test harness regression** — Issue #22307 reports that `PrefectRouter` no longer exposes a `routes` attribute in the version under test, breaking the documented `flow.serve()` test harness pattern. Until a fix lands, users hitting `AttributeError: 'PrefectRouter' object has no attribute 'routes'` should pin to a known-good release ([issue #22307](https://github.com/PrefectHQ/prefect/issues/22307)).
- **Recursion in parameter serialization** — 3.7.5.dev2 fixed a `RecursionError` in `Flow.serialize_parameters` that could surface when a deployment was created with deeply nested or self-referential parameters ([3.7.5.dev2 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev2)).

## See Also

- [README.md](https://github.com/PrefectHQ/prefect/blob/main/README.md) — project overview and quick links
- [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py) — full router composition
- [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py) — job variable validation rules
- [3.7.5 release notes](https://github.com/PrefectHQ/prefect/releases/tag/3.7.5) — current stable with worker/cleanup improvements

---

<a id='page-4'></a>

## Concurrency, Events, and Common Failure Modes

### Related Pages

Related topics: [System Architecture: Server, API, Database, and Events Pipeline](#page-2)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [src/prefect/server/api/server.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/server.py)
- [src/prefect/server/api/__init__.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/__init__.py)
- [src/prefect/server/api/concurrency_limits.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/concurrency_limits.py)
- [src/prefect/server/api/concurrency_limits_v2.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/concurrency_limits_v2.py)
- [src/prefect/server/api/run_history.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/run_history.py)
- [src/prefect/server/api/block_schemas.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_schemas.py)
- [src/prefect/server/api/block_documents.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/block_documents.py)
- [src/prefect/server/api/deployments.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/deployments.py)
- [src/prefect/server/api/validation.py](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/api/validation.py)
- [ui-v2/src/components/schemas/readme.md](https://github.com/PrefectHQ/prefect/blob/main/ui-v2/src/components/schemas/readme.md)

</details>

# Concurrency, Events, and Common Failure Modes

## Overview

Prefect exposes three operational pillars that frequently interact in production: a **concurrency limit system** (V1 adapter + V2 native), an **event streaming subsystem** used for automations and lifecycle observability, and a set of **recurring failure modes** that practitioners encounter when self-hosting or running flows concurrently. This page documents how these systems are wired in the server, what guarantees they offer, and the failure cases the community has surfaced.

The Prefect server assembles its public HTTP surface from a long list of FastAPI routers registered in `src/prefect/server/api/server.py`. Source: [src/prefect/server/api/server.py]() shows that `api.concurrency_limits.router`, `api.events.router`, `api.automations.router`, and `api.deployments.router` are all mounted alongside the UI routers. Source: [src/prefect/server/api/__init__.py]() enumerates the full module list, including the new `concurrency_limits_v2` namespace that supersedes V1 in newer releases.

## Concurrency System (V1 Adapter + V2 Native)

The V1 `/concurrency_limits` endpoints still exist but are implemented as a thin shim over the V2 system. Source: [src/prefect/server/api/concurrency_limits.py]() documents that the V1 router "routes requests to the V2 concurrency system. After the migration, V1 limits are converted to V2, but the V1 API continues to work for backward compatibility." V1 clients cannot renew leases, so the adapter issues a synthetic long-lived lease (`V1_LEASE_TTL = timedelta(days=100 * 365)`) so existing client code keeps working.

Source: [src/prefect/server/api/concurrency_limits_v2.py]() defines the V2 native model. It enforces that `slots` requested never exceeds the configured `limit`, rejects rate-limit mode usage for non-decaying limits (`slot_decay_per_second == 0.0`), and returns `MinimalConcurrencyLimitResponse` and `ConcurrencyLimitWithLeaseResponse` payload models. The two systems can be compared as follows:

| Aspect | V1 (`concurrency_limits`) | V2 (`concurrency_limits_v2`) |
|---|---|---|
| Lease renewal | Not supported by clients; long TTL applied | First-class lease lifecycle |
| Storage | V1 table, converted on read | Native lease storage backend |
| Rate limiting | Not supported | Supported via `slot_decay_per_second` |
| Backward compatibility | Yes | Authoritative |

The `_acquire_concurrency_slots()` helper in `concurrency_limits_v2.py` performs the active-limit filtering and raises `HTTPException(422)` if the requested slot count exceeds the limit. Source: [src/prefect/server/api/concurrency_limits_v2.py]().

## Events and Lifecycle Observability

The 3.7.4 release introduced lifecycle events (`created`, `updated`, `deleted`) for almost every domain object — variables, flows, block documents, block types, concurrency limits, artifact collections, and automations. These events drive the automations router and feed UI views of resource state changes. Source: [src/prefect/server/api/server.py]() mounts `api.events.router` and `api.automations.router` for this surface.

Ephemeral event subscriptions (used by the WebSocket events feed) rely on Redis consumer groups when `PREFECT_MESSAGING_BROKER=prefect_redis.messaging`. The community has tracked a defect (Issue #22136) where ungraceful shutdowns leak `ephemeral-{hostname}` consumer groups that pin the `events` stream and prevent trimming — a failure mode operators must plan for when self-hosting Redis.

The UI also consumes server-rendered JSON schemas for form generation. Source: [ui-v2/src/components/schemas/readme.md]() explains that `SchemaForm` validates values server-side via `validateSchemaValues` and routes by `__prefect_kind`, which is how block-document edits, deployment parameter edits, and work-pool configuration all funnel through a single component.

## Common Failure Modes

### 1. Concurrent subflow result collisions
Issue #22259 reports that when subflows are submitted concurrently via `task.submit()` (used because Prefect 3 lacks `flow.map()`), the virtual task run that tracks each subflow receives a positional `dynamic_key` from a per-flow-run counter. On parent retry, two subflow virtual tasks can collide on the same `dynamic_key` and a subflow will pick up the **other** subflow's persisted result. Source: [src/prefect/server/api/deployments.py]() shows the server-side flow-run creation path that produces these virtual tasks.

### 2. Test-harness `PrefectRouter.routes` AttributeError
Issue #22307 documents a broken test-harness path: `AttributeError: 'PrefectRouter' object has no attribute 'routes'`. The `PrefectRouter` used by every API module (e.g. `src/prefect/server/api/concurrency_limits.py`, `src/prefect/server/api/block_schemas.py`) wraps FastAPI's `APIRouter`, so test code that touches the FastAPI default attribute fails until the test fixture is updated to use the wrapper API.

### 3. Stuck `CANCELLING` runs
Release 3.7.5 ("No run left CANCELLING") added a cancelling-timeout cleanup producer (PR #22238) and backend observability for the worker channel and cleanup queue (PR #22310). Source: [src/prefect/server/api/run_history.py]() is the aggregation endpoint used by the UI to display these states across `flow_run` and `task_run` tables.

### 4. SQLite lock noise and FastAPI 0.137 compatibility
Source: [src/prefect/server/api/server.py]() installs `_SQLiteLockedOperationalErrorFilter` so that `SQLITE_BUSY` and "database is locked" errors are gated behind `PREFECT_SERVER_LOG_RETRYABLE_ERRORS` rather than spamming logs. Release 3.7.5.dev4 also patched FastAPI 0.137 route-deletion incompatibility.

### 5. Job-variable validation edge cases
Source: [src/prefect/server/api/validation.py]() documents an ordering rule that often surprises users: work-pool base template → deployment → flow run, with flow-run validation being the **only** step that enforces required fields. Invalid defaults in a work-pool template can silently survive deployment creation and only blow up when a flow run is actually launched.

### 6. Block document / schema conflicts
Source: [src/prefect/server/api/block_documents.py]() returns `409 Conflict` when a block with the same `(block_type_id, name)` already exists, and Source: [src/prefect/server/api/block_schemas.py]() deduplicates by `checksum + version`. Operators reusing block names across environments will hit the 409 unless they delete or rename first.

## See Also

- [Prefect REST API reference](https://docs.prefect.io/latest/api-ref/)
- Concurrency V2 migration guide
- Events and Automations concepts
- Self-hosting Redis messaging broker guidance
- Release notes for 3.7.4 and 3.7.5

---

<!-- evidence_pipeline_checked: true -->
<!-- evidence_injected: true -->

---

## Pitfall Log

Project: PrefectHQ/prefect

Summary: Found 24 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

## 1. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this installation risk before relying on the project: 3.7.4 - From cradle to grave
- User impact: Upgrade or migration may change expected behavior: 3.7.4 - From cradle to grave
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.4

## 2. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this installation risk before relying on the project: 3.7.4.dev3: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.4.dev3: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.4.dev3

## 3. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/PrefectHQ/prefect/issues/22259

## 4. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/PrefectHQ/prefect/issues/22136

## 5. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/PrefectHQ/prefect/issues/22276

## 6. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.4.dev4: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.4.dev4: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.4.dev4

## 7. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5 - No run left CANCELLING
- User impact: Upgrade or migration may change expected behavior: 3.7.5 - No run left CANCELLING
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5

## 8. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev1: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev1: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev1

## 9. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev2: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev2: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev2

## 10. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev3: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev3: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev3

## 11. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev4: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev4: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev4

## 12. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev5: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev5: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev5

## 13. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: 3.7.5.dev6: Nightly Development Release
- User impact: Upgrade or migration may change expected behavior: 3.7.5.dev6: Nightly Development Release
- Evidence: failure_mode_cluster:github_release | https://github.com/PrefectHQ/prefect/releases/tag/3.7.5.dev6

## 14. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this configuration risk before relying on the project: Leaked `ephemeral-*` Redis consumer groups pin the `events` stream and prevent trimming after ungraceful shutdowns
- User impact: Developers may misconfigure credentials, environment, or host setup: Leaked `ephemeral-*` Redis consumer groups pin the `events` stream and prevent trimming after ungraceful shutdowns
- Evidence: failure_mode_cluster:github_issue | https://github.com/PrefectHQ/prefect/issues/22136

## 15. Capability evidence risk - Capability evidence risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/PrefectHQ/prefect

## 16. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Developers should check this runtime risk before relying on the project: Broken test harness due to missing routes attribute in PrefectRouter
- User impact: Developers may hit a documented source-backed failure mode: Broken test harness due to missing routes attribute in PrefectRouter
- Evidence: failure_mode_cluster:github_issue | https://github.com/PrefectHQ/prefect/issues/22307

## 17. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/PrefectHQ/prefect/issues/22307

## 18. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/PrefectHQ/prefect/issues/22334

## 19. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/PrefectHQ/prefect

## 20. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/PrefectHQ/prefect

## 21. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://github.com/PrefectHQ/prefect

## 22. Capability evidence risk - Capability evidence risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: Developers should check this capability risk before relying on the project: Concurrent subflow calls adopt the WRONG subflow's persisted result on parent flow retry (positional dynamic_key race)
- User impact: Developers may hit a documented source-backed failure mode: Concurrent subflow calls adopt the WRONG subflow's persisted result on parent flow retry (positional dynamic_key race)
- Evidence: failure_mode_cluster:github_issue | https://github.com/PrefectHQ/prefect/issues/22259

## 23. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/PrefectHQ/prefect

## 24. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/PrefectHQ/prefect

<!-- canonical_name: PrefectHQ/prefect; human_manual_source: deepwiki_human_wiki -->