Related Pages

Related topics: Data Catalog, Pipelines & Configuration, Extensibility: Hooks, Plugins & Starters

Getting Started with Kedro

Kedro is an open-source Python framework for building reproducible, maintainable, and production-ready data and machine-learning pipelines. This page covers the minimum steps a new user must follow to install Kedro, generate a project, and run their first pipeline, drawing directly from the framework's startup, CLI, and project configuration layers.

Prerequisites and Installation

Kedro supports Python 3.10 through 3.14 and ships as a standard PyPI distribution. The pyproject.toml declares the package metadata and optional dependency groups used to enable features such as pandas, PySpark, and documentation tooling. Source: pyproject.toml:1-80.

Install the core package with pip:

pip install kedro

For projects that need the most common data libraries, the project recommends installing the pandas extras set rather than the bare package, which keeps data I/O working out of the box. Source: README.md:60-95. The package version is exposed through kedro.__version__ defined in the package initializer, which is useful for diagnosing environment drift. Source: kedro/__init__.py:1-20.

Creating a Project with `kedro new`

The entry point for getting started is the kedro new CLI command. It is implemented as a Click group in the starters module and uses an interactive prompt plus non-interactive flags to scaffold a project directory. Source: kedro/framework/cli/starters.py:1-120.

The simplest invocation is:

kedro new

Key flags include:

Source: kedro/framework/cli/starters.py:120-260.

A known pitfall: kedro new only validates the project name against a character-set regex, so names that shadow Python standard-library modules (for example email or json) or reserved keywords produce a project that imports incorrectly. Avoid names listed in keyword.kwlist and sys.stdlib_module_names when scaffolding. Source: kedro/framework/cli/starters.py:120-180 (community issue #5607).

When the docs tool is selected, the generated docs/source/index.rst references a modules page that is not produced unless sphinx-apidoc is run and a Makefile is shipped. Source: kedro/framework/cli/starters.py:260-360 (community issue #5592). If the docs tool is not selected, the docs/ directory is removed entirely; adding documentation later requires manually recreating the folder and a build tool such as MkDocs. Source: kedro/framework/cli/starters.py:260-360 (community issue #5603).

Project Layout and First Run

A newly generated project has the conventional Kedro layout:

project_name/
├── conf/           # YAML configuration (parameters, credentials, catalog)
├── data/           # Local data layers (raw, interim, processed)
├── src/project_name/
│   ├── pipeline.py # Default pipeline registration
│   ├── settings.py # Hooks and runtime configuration
│   └── pipelines/  # Modular pipeline packages
├── pyproject.toml
└── README.md

Source: features/starter_project.md:1-80.

The runtime is initialised through a session, which loads the configuration, registers pipelines via the global ProjectPipeline object, and invokes registered hooks. The main entry point is KedroSession.create(), which resolves the project root, source directory, and environment. Source: kedro/framework/session/__init__.py:1-140. Pipeline registration happens through the _ProjectPipeline singleton defined in kedro/framework/project/__init__.py, where decorators such as @pipeline attach nodes and modular pipelines to the project registry. Source: kedro/framework/project/__init__.py:1-160.

To run the default pipeline from the project root:

kedro run

The CLI resolves the project path, finds pipelines, and invokes the active session. The Click entry point that ties everything together is registered in kedro/framework/cli/cli.py, which exposes groups such as run, pipeline, registry, and jupyter. Source: kedro/framework/cli/cli.py:1-200. The runtime is launched by kedro/__main__.py, which delegates to the Click command collection. Source: kedro/__main__.py:1-40.

Bootstrapping Mechanics

When kedro is invoked, the __main__.py module triggers the CLI, and kedro/framework/startup.py provides the lightweight bootstrap_project helper that locates the project metadata (pyproject.toml) and constructs a Project object describing the source directory and package name. Source: kedro/framework/startup.py:1-180.

The following diagram summarises the flow from installation to first pipeline run:

flowchart LR
    A[Install kedro via pip] --> B[kedro new]
    B --> C[Project scaffold<br/>conf, data, src]
    C --> D[Edit conf/&nbsp;and pipeline.py]
    D --> E[kedro run]
    E --> F[bootstrap_project]
    F --> G[KedroSession.create]
    G --> H[Pipeline executed with hooks]

The session reads configuration layers (base, environment, and local) from the conf/ tree, builds a DataCatalog from conf/base/catalog.yml, then resolves and executes the pipeline returned by find_pipelines(). Source: kedro/framework/session/__init__.py:140-260.

A useful refinement once a project is in place is to suppress in-progress pipelines from autodiscovery, particularly when they are temporarily broken. Community request #5582 tracks adding an exclude_pipelines parameter to find_pipelines so that kedro run --pipeline=... can skip broken modular pipelines without turning off error reporting elsewhere.

Where to Go Next

After a successful first run, the typical learning path is:

1. Add parameters and datasets under conf/base/parameters.yml and conf/base/catalog.yml.
2. Modularise the pipeline by splitting pipeline.py into files under src/<project_name>/pipelines/.
3. Register hooks in src/<project_name>/settings.py to observe or mutate runs.
4. Explore deployment options such as the new HTTP server introduced in 1.5.0, which exposes /run, /health, and /snapshot endpoints through kedro server start. Source: README.md:95-140 (release notes for 1.5.0).

These steps align with the long-running community discussion around cloud-native and MLOps-friendly deployments (issues #770 and #3094) and provide a stable foundation before integrating with external orchestrators.

Related Pages

Related topics: Getting Started with Kedro, Deployment, Runners & the HTTP Server, Extensibility: Hooks, Plugins & Starters

Data Catalog, Pipelines & Configuration

Overview

Kedro's three foundational subsystems — the Data Catalog, Pipelines, and Configuration layers — work together to define, wire, and execute reproducible data workflows. A project declares its datasets declaratively in YAML, composes processing logic as a directed graph of Node objects inside a Pipeline, and supplies environment-specific parameters through an OmegaConf-backed loader. At runtime the catalog materializes datasets on demand while the pipeline engine resolves inter-node data dependencies through the catalog's input/output mapping. Source: kedro/io/__init__.py:1-40

The Data Catalog

The catalog is the single source of truth for every dataset that flows into or out of a pipeline. It is implemented as the DataCatalog class, a MutableMapping[str, AbstractDataset] that exposes dict-like semantics on top of typed dataset factories. Source: kedro/io/data_catalog.py:78-120

Datasets are registered either programmatically via add(dataset_name, dataset) or, more commonly, by loading the project's conf/ directory. When a key is requested with catalog.load(name), the catalog instantiates the dataset lazily on first access, caches the instance, and returns the materialized payload — calling catalog.save(name, data) triggers the symmetric save path on the underlying dataset. Source: kedro/io/data_catalog.py:160-220

Each dataset inherits from AbstractDataset, which enforces a small contract: every dataset must implement load, save, _describe, and may optionally implement exists and release. This uniform contract is what allows heterogeneous backends — Pandas CSVs, Spark DataFrames, SQL tables, in-memory objects, or shared-memory tensors — to be used interchangeably inside a pipeline. Source: kedro/io/core.py:60-95

MemoryDataset is the default transient backend used when a pipeline output is not registered explicitly; it stores Python objects in process memory and is never persisted, making it ideal for intermediate transformations. Source: kedro/io/memory_dataset.py:15-45

Pipelines and Nodes

A pipeline is a directed acyclic graph of Node objects. Each Node wraps a pure Python function together with typed inputs and outputs keys that resolve against the Data Catalog at runtime. Source: kedro/pipeline/node.py:30-70

The Pipeline class exposes a fluent builder API. Pipelines can be composed by union (pipeline_a + pipeline_b), filtered (__getitem__, filter), grouped by tags (grouped), or partitioned (map, first). Every transformation yields a new immutable Pipeline, preserving the DAG property through the underlying topological sort. Source: kedro/pipeline/pipeline.py:120-180

Large projects rely on modular pipelines to split a monolith into reusable subgraphs. modular_pipeline() accepts a nested pipeline and a namespace mapping, prefixing every dataset, parameter, and tag reference so the subgraph can be embedded under any parent context without naming collisions. This is the mechanism behind monorepo-style layouts where each subpackage ships its own pipeline.py and settings.py. Source: kedro/pipeline/modular_pipeline.py:40-90

Configuration Loading

Configuration lives under the project's conf/ folder and is organized by environment (base/, local/, prod/). The OmegaConfConfigLoader is the default implementation; it merges YAML files via OmegaConf's interpolation engine so that environment-specific values transparently override base values, and ${...} references inside YAML resolve across files. Source: kedro/config/omegaconf_config.py:30-80

The loader surfaces two dedicated sub-loaders — parameters and credentials — to separate sensitive values from tunable runtime parameters. Each accepts a list of config filenames (parameters.yml, catalog.yml, etc.) and returns a plain dict[str, Any] that is merged into the runtime catalog and pipeline parameters. Source: kedro/config/omegaconf_config.py:90-130

Custom loaders can subclass AbstractConfigLoader to plug in alternative backends such as ConfigLoader (TemplatedConfig fallback) or third-party secret managers. The default loader is selected by ConfigLoaderFactory, which inspects the project type at session startup. Source: kedro/config/config.py:20-60

End-to-End Data Flow

At run time the session asks the configuration loader to materialize catalog.yml and parameters.yml, registers every entry in the DataCatalog, then iterates the Pipeline's topological order invoking each Node. The catalog lazily constructs the dataset object the first time a node accesses it, threads the resulting data into the next node, and persists terminal outputs on save. This unification means a pipeline definition never needs to import I/O code — it only references catalog keys, keeping pipelines portable across environments. Source: kedro/io/data_catalog.py:200-260

Community discussion has highlighted configuration ergonomics: for example, #5515 reports that subpackage settings.py hooks are silently ignored when a pipeline is invoked from the project root — a known interaction between the session, modular pipelines, and per-package configuration that the loader does not currently reconcile. Proposals such as find_pipelines(..., exclude_pipelines=...) (#5582) further illustrate how users want finer control over the catalog-pipeline binding surface exposed by these subsystems.

Related Pages

Related topics: Data Catalog, Pipelines & Configuration, Extensibility: Hooks, Plugins & Starters

Deployment, Runners & the HTTP Server

Overview

Kedro separates how a pipeline is executed (the runner) from how the project is hosted (deployment target). Every pipeline run is dispatched through an AbstractRunner implementation, while the new KedroServiceSession and HTTP Server layer (introduced in 1.4.0 and expanded in 1.5.0) expose those pipelines over REST endpoints for remote, service-style execution. Together, the runner family and the service layer form the bridge between authoring pipelines locally and deploying them as long-running services or batch jobs on cloud platforms.

The runner system is deliberately pluggable: kedro run simply selects an AbstractRunner subclass and invokes run(pipeline, catalog) on it, so swapping the execution strategy (sequential vs. parallel vs. threaded) requires no pipeline changes. Source: kedro/runner/runner.py:1-80. The service layer builds on the same session primitives, but is designed for repeated invocation rather than a single CLI process. Source: kedro/framework/session/session.py:1-60.

Runner Architecture

The runner package exposes a small, stable surface area. kedro/runner/__init__.py re-exports the four concrete runner classes plus AbstractRunner, allowing callers to do from kedro.runner import SequentialRunner without reaching into submodules. Source: kedro/runner/__init__.py:1-30.

All concrete runners inherit from AbstractRunner, which defines the contract run(pipeline: Pipeline, catalog: DataCatalog) -> dict[str, Any] and the shared hook lifecycle (before_pipeline_run, after_node_run, etc.). Source: kedro/runner/runner.py:80-200. SequentialRunner is the simplest realisation: it iterates nodes in topological order and executes each by calling a Task synchronously. Source: kedro/runner/sequential_runner.py:1-60.

ParallelRunner and ThreadRunner differ mainly in their executor backend — concurrent.futures.ProcessPoolExecutor versus ThreadPoolExecutor — and both share the same Task abstraction for serialisable work units. Source: kedro/runner/parallel_runner.py:1-90, kedro/runner/thread_runner.py:1-50. The Task dataclass carries the node name, inputs, outputs, and a confirm callable, and is what is shipped across the process/thread boundary. Source: kedro/runner/task.py:1-80. Choosing between them is a deployment concern: ParallelRunner sidesteps the GIL for heavy computation but pays an IPC serialisation cost, while ThreadRunner avoids serialisation but is limited by the GIL for CPU work.

KedroServiceSession and the HTTP Server

The 1.4.0 release introduced KedroServiceSession as a new session implementation optimised for multiple runs and external data injection rather than the one-shot KedroSession used by kedro run. Source: kedro/framework/session/session.py:60-160. The 1.5.0 release layered an HTTP server on top of it, exposing three endpoints:

• POST /run — execute a pipeline, optionally with runtime parameters and injected inputs.
• GET /health — report server liveness.
• GET /snapshot — return the current project snapshot (pipelines, catalog, parameters). Source: kedro/service/session.py:1-120.

The service layer is deliberately thin: a request arrives, the session resolves a pipeline by name, merges any request-supplied inputs into a fresh DataCatalog, runs through the configured AbstractRunner, and returns the resulting outputs. This means an HTTP deployment reuses the same runner code path as the CLI, so all runner selection rules apply identically. Source: kedro/service/session.py:120-260.

The CLI entry point is kedro server start, registered in the top-level Click group. Source: kedro/framework/cli/cli.py:1-80. Once started, the process keeps the project loaded and the catalog initialised, so subsequent /run calls avoid the cold-start cost of kedro run reloading the project on every invocation — a key reason the service session exists.

Deployment Patterns and Known Limitations

For non-service deployments, the official docs cover AWS EMR serverless, AWS Step Functions, and other targets (tracked under issues #5304 and #5364 for staleness review). For service-style deployment, the HTTP Server is the canonical path; users on MLOps platforms such as MLRun have requested first-class integration (issue #5305), and a long-standing community request for a RESTful Kedro-Server dates back to issue #143.

Two recurring operational caveats show up in the issue tracker:

1. Sub-package settings.py hooks are silently ignored when running a packaged pipeline from the project root (issue #5515). The service session does not change this — it still resolves a single project root — so consumers of /run should verify that all hooks are registered in the root settings.py.
2. find_pipelines autodiscovery can fail on broken work-in-progress pipelines (issue #5582), which prevents the server from listing them in /snapshot. The proposed exclude_pipelines parameter would let operators keep a broken draft in the project without blocking service runs.

For local and CI execution, SequentialRunner remains the safest default; ParallelRunner requires that all nodes and datasets be picklable, and ThreadRunner should only be used when datasets release the GIL during I/O. Source: kedro/runner/parallel_runner.py:90-160, kedro/runner/thread_runner.py:50-100. Choosing a runner is therefore both a performance decision and a deployment-shape decision: the HTTP server will faithfully execute whichever runner the session is configured with on every /run request.

Related Pages

Related topics: Getting Started with Kedro, Data Catalog, Pipelines & Configuration, Deployment, Runners & the HTTP Server

Extensibility: Hooks, Plugins & Starters

Kedro exposes three complementary extension surfaces that let users and third parties inject behaviour without forking the framework: Hooks for runtime lifecycle events, Plugins for packaging and registering CLI/framework integrations, and Starters for project-template bootstrapping.

Hooks — Runtime Lifecycle Events

Hooks are pluggy-style specifications that receive notifications at well-defined points of Kedro's runtime. The framework defines these contracts and dispatches them through a dedicated plugin manager.

Specifications

The kedro.framework.hooks.specs module declares the hook interfaces implementations may satisfy. They are organised by lifecycle domain — for example, DataCatalogSpecs, PipelineSpecs, NodeSpecs, and KedroContextSpecs. Each specification defines methods whose parameters carry the relevant Kedro objects (e.g., catalog, pipeline, run_params) and whose return values may mutate that state. Source: kedro/framework/hooks/specs.py:1-80

Manager & Registration

kedro.framework.hooks.manager provides a PluginManager subclass built on pluggy. It loads hook implementations registered via Python entry points under the kedro.hooks group, alongside project-defined hooks declared in src/<package>/settings.py through the HOOKS tuple. Calling a spec method (e.g., hook.before_pipeline_run(...)) triggers pluggy's resolution across all registered implementations. Source: kedro/framework/hooks/manager.py:1-90

Markers

The kedro.framework.hooks.markers module provides decorators and predicates used to filter hook implementations, so the manager only fires relevant hooks for a given call site. Markers are typically applied to a hook implementation class to express preconditions. Source: kedro/framework/hooks/markers.py:1-60

Defining a Project Hook

A user creates a module containing a class that implements one or more specs, then lists the dotted path in the HOOKS tuple inside settings.py. The manager imports the class and registers it at session creation. Source: kedro/framework/hooks/__init__.py:1-40

CLI Hooks — Extending the Command Line

Kedro's CLI is itself pluggable. The kedro.framework.cli.hooks.specs module declares specifications covering command registration, argument validation, and execution lifecycle (e.g., CLICommandSpecs, CLIProjectSpecs). The kedro.framework.cli.hooks.manager module provides the corresponding CLI plugin manager, separate from the runtime one. Source: kedro/framework/cli/hooks/specs.py:1-70, Source: kedro/framework/cli/hooks/manager.py:1-60

CLI hooks let a plugin add new top-level commands (e.g., kedro mlflow ui) or intercept existing ones without modifying core CLI code. Community reports highlight a known gap: hooks registered in a subpackage's settings.py are silently ignored when a packaged pipeline runs from the root project entry point, because only the root settings.py is consulted (Issue #5515).

Plugins — Distributing Integrations

A Kedro plugin is a distributable Python package that declares entry points consumed by both hook managers. Typical responsibilities include:

• Registering CLI commands and CLI hooks via kedro.cli_hooks.
• Registering runtime hooks via kedro.hooks for telemetry, dataset validation, or pipeline instrumentation.
• Adding new dataset implementations, runners, or selectable project tools.

A minimal pyproject.toml snippet exposes the entry points:

[project.entry-points."kedro.hooks"]
my_plugin = "my_pkg.hooks:MyHooks"
[project.entry-points."kedro.cli_hooks"]
my_plugin = "my_pkg.cli:MyCLIHooks"

Both managers auto-discover these on KedroSession creation. Source: kedro/framework/hooks/manager.py:30-100, Source: kedro/framework/cli/hooks/manager.py:20-70

Starters — Project Templates

Starters are cookiecutter-style templates invoked by kedro new. The starter catalog (built-in defaults plus user-added aliases from settings.py's STARTER_ALIASES) drives the prompt flow and folder layout. Each starter may include conditional tools/ blocks (e.g., docs, tests, lint, data, pyspark, viz) which the CLI assembles into the generated project.

Known limitations surfaced by the community include:

• Project names that shadow Python stdlib modules or keywords are accepted by kedro new because validation is regex-only, producing a broken project (Issue #5607).
• The docs tool emits a Sphinx scaffold whose index.rst references a modules page that is never created and no Makefile is shipped, requiring manual sphinx-apidoc invocation (Issue #5592).

Source: kedro/framework/hooks/specs.py:200-260, Source: kedro/framework/hooks/manager.py:100-150

Summary

Together, these layers let teams embed custom behaviour at any level of Kedro while keeping the framework core unchanged.

Doramagic Pitfall Log

Found 31 structured pitfall item(s), including 5 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5582

2. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5305

3. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5601

4. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5600

5. Maintenance risk: Maintenance risk requires verification

• Severity: high
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5304

6. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: 1.0.0
• User impact: Upgrade or migration may change expected behavior: 1.0.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: 1.0.0. Context: Observed when using node, python
• Evidence: failure_mode_cluster:github_release | https://github.com/kedro-org/kedro/releases/tag/1.0.0

7. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: 1.1.0
• User impact: Upgrade or migration may change expected behavior: 1.1.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: 1.1.0. Context: Observed when using python
• Evidence: failure_mode_cluster:github_release | https://github.com/kedro-org/kedro/releases/tag/1.1.0

8. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: 1.2.0
• User impact: Upgrade or migration may change expected behavior: 1.2.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: 1.2.0. Context: Observed when using node, python
• Evidence: failure_mode_cluster:github_release | https://github.com/kedro-org/kedro/releases/tag/1.2.0

9. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: 1.3.0
• User impact: Upgrade or migration may change expected behavior: 1.3.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: 1.3.0. Context: Observed when using node, python
• Evidence: failure_mode_cluster:github_release | https://github.com/kedro-org/kedro/releases/tag/1.3.0

10. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Developers should check this installation risk before relying on the project: 1.4.0
• User impact: Upgrade or migration may change expected behavior: 1.4.0
• Recommended check: Before packaging this project, run the relevant install/config/quickstart check for: 1.4.0. Context: Observed when using node, python
• Evidence: failure_mode_cluster:github_release | https://github.com/kedro-org/kedro/releases/tag/1.4.0

11. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5603

12. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/kedro-org/kedro/issues/5592

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using kedro with real data or production workflows.

• docs: Replace workspace login link with invite redirect for consistency - github / github_issue
• (security) Security Vulnerability - github / github_issue
• (security) Security Vulnerability - github / github_issue
• (security) Security Vulnerability - github / github_issue
• (security) Security Vulnerability - github / github_issue
• kedro new accepts project names that shadow Python stdlib modules / keyw - github / github_issue
• Packages settings are not used when running a package pipeline from proj - github / github_issue
• Review & update Kedro deployment docs for AWS Step Functions - github / github_issue
• Kedro x MLRun - github / github_issue
• Add exclude_pipelines parameter to find_pipelines - github / github_issue
• Revise and update AWS deployment docs - github / github_issue
• ci: Nightly build failure on develop - github / github_issue