Related Pages

Related topics: Ray Data: Scalable Data Processing for ML, AI Libraries: Train, Tune, RLlib, Serve, and LLM, Deployment, Infrastructure, Observability, and Cross-Language Extensibility

Ray Overview and Core Distributed Runtime

Purpose and Scope

Ray is a distributed execution framework that lets Python programs seamlessly parallelize work across one or many machines. The top-level package exposes decorators, helpers, and the init/shutdown entry points used by every driver and worker. ray.init() connects a Python process to a Ray cluster (local or remote), configures runtime resources, and starts a background worker thread that talks to the local raylet over a Unix domain socket Source: python/ray/__init__.py:1-120. Every node participating in a Ray program runs the same core components regardless of role: a raylet process that schedules work on that node, a gcs_server (Global Control Store) that holds cluster-wide metadata, and one or more worker processes that host Python user code Source: doc/source/ray-core/key-concepts.rst:1-90. The framework is intentionally layered: high-level libraries (Ray Data, Ray Train, Ray Serve, Ray RLlib) all sit on top of the same two primitives — *tasks* and *actors* — exposed by Ray Core Source: doc/source/ray-core/walkthrough.rst:1-60.

System Architecture

A Ray cluster is composed of a single head node and zero or more worker nodes. Every node runs two daemons:

• A raylet, written in C++, which manages local scheduling, the plasma object store, and worker lifecycle Source: src/ray/raylet/raylet.cc:1-80.
• (Head-only) A gcs_server, which stores cluster metadata such as node liveness, actor locations, and job state Source: doc/source/ray-core/key-concepts.rst:90-160.

User code runs inside *worker* processes spawned on demand by the local raylet. Each worker embeds a Cython bridge (_raylet.pyx) that marshals task submissions, object references, and actor method calls into gRPC messages sent to the raylet Source: python/ray/_raylet.pyx:1-70. The driver is just a special worker that happens to host the user's top-level script Source: python/ray/_private/worker.py:1-120.

flowchart LR
    Driver[Driver / Python script] -->|gRPC| Raylet1[Head raylet]
    Raylet1 --> GCS[(GCS<br/>cluster metadata)]
    Worker1[Worker process] --> Raylet1
    Raylet2[Worker node raylet] --> GCS
    Worker2[Worker process] --> Raylet2
    Raylet1 <-->|heartbeat /<br/>task forward| Raylet2
    Raylet2 -.->|plasma| Plasma[(Object store<br/>plasma)]

The two long-lived cross-node channels are the GCS for control-plane data and the plasma object store (and, more recently, the Ray Data/DataContext shared-memory paths) for bulk data transfer between workers on different nodes Source: src/ray/common/task/task_spec.h:1-90.

Core Abstractions

Tasks

A *task* is a stateless function invocation marked with @ray.remote. Calling f.remote(x) schedules the function on a worker and immediately returns an ObjectRef instead of a concrete value Source: python/ray/remote_function.py:1-120. Each task is described by a TaskSpec protobuf that carries the function ID, arguments (as ObjectRefs or values), resource requirements, and placement hints Source: src/ray/common/task/task_spec.h:90-180. The local raylet queues the task, places it on a worker that satisfies its CPU/GPU/accelerator demands, and runs it; results are written to plasma and the returned ObjectRef becomes resolvable with ray.get() Source: doc/source/ray-core/tasks.rst:1-80.

Actors

An *actor* is a stateful worker bound to a class instance. @ray.remote on a class produces an ActorClass, and .remote() spawns the actor on a chosen node and returns an ActorHandle Source: python/ray/actor.py:1-150. Unlike tasks, actor methods execute serially on the actor's dedicated worker; the raylet guarantees that all method calls to a given actor land on the same process, and that the actor's lifetime is independent of any single caller Source: doc/source/ray-core/key-concepts.rst:160-240. Actor handles can be passed across tasks and other actors, which is how Ray implements distributed shared state.

Objects and the Object Store

Any Python value wrapped with ray.put(v) is serialized and stored in the local plasma object store, returning an immutable ObjectRef Source: python/ray/_private/worker.py:120-220. Object refs are first-class values: they can be returned from tasks, stored on actors, and passed as arguments to other tasks. The runtime resolves transitive dependencies through ObjectRef IDs, so downstream tasks do not block until upstream results are materialized — a property the scheduler exploits for pipelining Source: doc/source/ray-core/tasks.rst:80-160.

Execution Flow and Lifecycle

When a driver calls f.remote(args):

1. The Cython bridge constructs a TaskSpec and posts it to the local raylet Source: python/ray/_raylet.pyx:70-160.
2. The raylet resolves argument ObjectRefs (fetching from plasma if they are remote), picks a worker satisfying the task's resource request, and dispatches the task Source: src/ray/raylet/raylet.cc:80-200.
3. The worker executes the Python function, writes the return value to plasma, and replies to the raylet with a new ObjectRef Source: python/ray/_private/worker.py:220-340.
4. ray.get(ref) blocks on the driver until the corresponding object is available, deserializing it lazily Source: python/ray/_private/worker.py:340-440.

ray.shutdown() reverses this by detaching the worker from the local raylet, draining in-flight tasks, and tearing down the runtime context; in cluster mode it does not kill remote workers owned by other drivers Source: python/ray/_private/worker.py:440-540. The same plumbing is what underpins Ray 2.x's stability push — issues like #54923 (Q3 2025 roadmap) explicitly call out hardening the core task and object lifecycles documented above as a foundation for higher-level libraries.

Related Pages

Related topics: Ray Overview and Core Distributed Runtime, AI Libraries: Train, Tune, RLlib, Serve, and LLM

Ray Data: Scalable Data Processing for ML

Ray Data is a distributed data processing library built on top of Ray Core that provides a scalable API for loading, transforming, and consuming large datasets in machine learning workloads. It is designed to handle data ingestion, preprocessing, and shuffling across a Ray cluster while integrating tightly with Ray Train and Ray Serve for end-to-end ML pipelines. The library is one of the major Ray 2.0 focus areas called out in the Ray 2.0 RFC (issue #22833), and stability improvements such as multi-dataset execution, automatic batch size selection for CPU map-batches, and default logical memory configuration are highlighted in the Ray 2.56.0 release notes.

Architecture Overview

Ray Data separates the user-facing Dataset API from its internal execution and optimization layers. The public entry point in python/ray/data/__init__.py exposes high-level constructors (range, range_table, from_items, from_arrow, from_pandas, from_spark, read_*, read_datasource) and transformations (map, map_batches, flat_map, filter, groupby, join, sort, random_shuffle, repartition, split, union, limit, zip). These compose into a lazy logical plan that is only resolved when a consuming operation such as iter_batches, iter_rows, to_pandas, to_arrow, to_tf, to_torch, write_*, count, show, or stats is called. Source: python/ray/data/__init__.py.

Internally, Dataset wraps a LogicalOperator DAG, a Schema, and an Expression subsystem. Transformations append new operators; reading sources returns a FromX operator that produces a Block stream. The LogicalPlan is passed through a LogicalOptimizer (defined in python/ray/data/_internal/logical/optimizers.py) which runs a sequence of registered rewrite rules before physical planning, enabling fusion, projection pushdown, and operator reordering. Source: python/ray/data/_internal/logical/optimizers.py.

flowchart LR
    A[read_* / from_*] --> B[LogicalPlan]
    C[map / map_batches / filter / groupby / ...] --> B
    B --> D[LogicalOptimizer + Rules]
    D --> E[StreamingExecutor]
    E --> F[iter_batches / to_torch / write_*]

Execution is driven by the streaming executor in python/ray/data/_internal/execution/streaming_executor.py, which converts operators into PhysicalOperator instances, manages a topology of Block queues, and schedules tasks onto Ray actors. The streaming model allows backpressure and overlapping I/O with compute; recent 2.56.0 changes explicitly mention reducing hidden buffering in iter_batches and shutting down the executor cleanly to prevent leaks. Source: python/ray/data/_internal/execution/streaming_executor.py.

Reading and Transforming Data

The read_api module is the canonical surface for ingesting data from files, databases, and ML-aware formats. It supports row-oriented sources via read_text and read_csv, columnar sources via read_parquet and read_arrow, ML formats via read_images, read_numpy, read_tfrecords, and read_mlflow, and external systems via read_databricks, read_datasource, read_unity_catalog, and read_snowflake. Each reader accepts a DataContext for resource hints, parallelism, and schema configuration, and returns a lazy Dataset. Source: python/ray/data/read_api.py.

Transformations follow a familiar functional style. Stateless row operations use map or flat_map; vectorized batch operations use map_batches, which lets user functions receive Arrow, Pandas, or a torch tensor batch and operate on whole columns. Stateful or shuffled operations such as groupby, sort, random_shuffle, repartition, and join are routed through dedicated logical operators that the optimizer can fuse with adjacent steps when possible. The Dataset class also exposes split, limit, union, zip, and column projections (select_columns, rename_columns, drop_columns) for shaping the pipeline. Source: python/ray/data/dataset.py.

Configuration and Execution Options

Runtime behavior is controlled through DataContext, defined in python/ray/data/context.py. It exposes knobs for the executor (execution_options), the default batch format (default_batch_format), shuffle strategy (shuffle_strategy), memory budgets (_memory_usage_pinning_*, _max_num_blocks_in_streaming_gen), and instrumentation such as enable_auto_log_stats and print_on_exception. Per-pipeline overrides can be applied by passing an ExecutionOptions value through Dataset.execution_options, which lets callers tweak resource_limits, actor_locality_enabled, preserve_order, and verbose_progress. Source: python/ray/data/context.py.

Optimizer rules live alongside LogicalOptimizer in python/ray/data/_internal/logical/rules/__init__.py. Rule registration, ordering, and fast-path checks are part of the public-ish optimizer API, and custom rule sets can be injected to experiment with new rewrites. The interaction between DataContext, ExecutionOptions, and rule selection determines how a logical plan is scheduled: for example, enabling shuffle fusion or restricting the number of concurrent operators affects how random_shuffle is fused with downstream readers. Source: python/ray/data/_internal/logical/rules/__init__.py.

Integration with Ray ML Ecosystem

Ray Data is positioned as the data layer for the rest of Ray ML. Dataset.iter_batches and Dataset.to_torch are designed to feed Ray Train's DataConfig.train_dataloader and DataConfig.valid_dataloader, and to_tf mirrors that path for TensorFlow. The streaming executor overlaps ingestion with the training step by prefetching batches ahead of the trainer. This design is exactly the one called out in the Ray 2.0 RFC (issue #22833) for higher-level ML APIs, and Q3 2025 roadmap issue #54923 lists continued Data stability work — multiple datasets on a cluster, automatic batch size selection, and default logical memory configuration — that shipped in 2.56.0. The framework also serves as input for Ray Serve online inference through to_pandas / iter_rows for feature pipelines.

Operational Considerations

Because execution is lazy, datasets can be inspected with schema(), count(), show(), stats(), and explain() without materializing them, which is helpful when debugging a long pipeline. Failures in streaming execution are surfaced through Dataset.errors() and per-block logs in the Ray Data logs page of the Ray dashboard, both of which rely on DataContext.enable_logging_to_driver and the executor's stats subscribers. Per the release notes for 2.56.0, the executor now aggressively tears down resources on iter_batches shutdown to prevent OOMs, and default logical memory bounds prevent the planner from over-subscribing the cluster. Source: python/ray/data/_internal/execution/streaming_executor.py.

Related Pages

Related topics: Ray Overview and Core Distributed Runtime, Ray Data: Scalable Data Processing for ML, Deployment, Infrastructure, Observability, and Cross-Language Extensibility

AI Libraries: Train, Tune, RLlib, Serve, and LLM

Ray provides a layered ecosystem of "AI Libraries" built on top of Ray Core. These libraries target distinct stages of the ML lifecycle — distributed training, hyperparameter tuning, reinforcement learning, model serving, and large language model inference — and share Ray's task/actor primitives for distributed execution.

Ray Train: Scalable Distributed Training

ray.train exposes a framework-agnostic, fault-tolerant training API across PyTorch, TensorFlow, and HuggingFace Transformers via backend integrations (TorchTrainer, TensorflowTrainer, HuggingFaceTrainer). ray.train re-exports top-level utilities such as train.report, train.Checkpoint, and the ScalingConfig/RunConfig dataclasses that drive backend-agnostic configuration Source: python/ray/train/__init__.py:1-50.

ray.train.v2 is the next-generation trainer entry point. It simplifies configuration by exposing a single TorchConfig and a unified trainer.fit() surface, replacing several disparate v1 entry points. The v2 controller (TrainController) is a long-running actor that owns the worker group lifecycle, fault handling, and shutdown logic Source: python/ray/train/v2/_internal/execution/controller/controller.py:1-80.

from ray.train.v2 import Trainer
from ray.train.torch import TorchConfig

trainer = Trainer(
    num_workers=4,
    use_gpu=True,
    backend=TorchConfig(),
    scaling_config=...,
)
trainer.fit(train_loop_per_worker=lambda: None)

Ray Tune: Hyperparameter Search and Experiment Orchestration

Ray Tune is the hyperparameter optimization (HPO) library within Ray. The legacy entry point ray.tune.run (defined in tune.py) remains a thin compatibility shim that delegates to the modern Tuner class Source: python/ray/tune/tune.py:1-40.

The modern API lives in Tuner, which decouples experiment specification from execution. Users compose a TuneConfig (defining the search space, scheduler, and metric) and pass a trainable (function or class) to Tuner.fit(). Scheduling algorithms such as ASHAScheduler, PopulationBasedTraining, and Bayesian search drivers are first-class citizens Source: python/ray/tune/tuner.py:1-60.

• Schedulers: early-stopping and population-based schedules.
• Search algorithms: random, grid, Optuna, Bayesian.
• Tune callbacks for integration with Train, RLlib, and external dashboards.

RLlib: Reinforcement Learning

RLlib (python/ray/rllib) is the reinforcement learning library. It is built as a self-contained stack of RL-specific algorithms (PPO, SAC, IMPALA, APPO, etc.) that inherit from a common Algorithm base class. Algorithms are configured via AlgorithmConfig objects; training is launched by calling algorithm.train() or via the tune.Tuner integration with rllib.trainable. RLlib co-locates with Tune so that RL hyperparameter sweeps reuse the search/scheduler machinery above.

Ray Serve: Online Model Serving

Ray Serve (python/ray/serve) is the model-serving library. It is built around the concept of a Deployment, a route in a Ray actor that exposes a @serve.deployment wrapped callable or class. The serve.run() (or for v2, serve.deploy() with an Application config) entry point materializes the deployment graph across the cluster.

Serve provides:

• Stateless and stateful deployments with autoscaling (configurable num_replicas, target_ongoing_requests, max_ongoing_requests).
• Fast HTTP ingress and gRPC adapters, plus Python request batching (@serve.batch).
• Multi-model composition via Ingress, Router, and DAG/handle APIs.
• Optimistic autoscaling in newer versions (an explicit Q3 2025 roadmap focus).

Ray LLM: LLM Inference and Fine-tuning

python/ray/llm is Ray's library for large language model serving, fine-tuning, and distributed inference. It wraps vLLM (and equivalent engines) inside Ray Serve deployments, exposing LLMServing configurations that orchestrate PlacementGroup-based prefill/decode disaggregation, LoRA adapters, and the v2 Serve config schemas. Fine-tuning pipelines are exposed via Ray Train backends and integrate with rayllm.tune for sweeps.

How the Libraries Fit Together

Community discussions (#22833, "Ray 2.0 Feature Proposals") highlighted unified scheduling APIs and ecosystem interoperability across these five libraries as core usability goals, and Ray's Q3 2025 roadmap (#54923) treats Train, Serve, RLlib, and LLM as separate but tightly-coupled delivery streams — each must keep its own reliability guarantees while sharing Ray's execution substrate.

Related Pages

Related topics: Ray Overview and Core Distributed Runtime, AI Libraries: Train, Tune, RLlib, Serve, and LLM

Deployment, Infrastructure, Observability, and Cross-Language Extensibility

Ray is more than its task and actor APIs; the production surface area of the project spans cluster deployment, autoscaling, runtime observability, and polyglot extension points. This page surveys those subsystems, what role each plays, and how they are wired together in the source tree. The scope covers everything a user or operator needs to provision a cluster, scale it dynamically, observe its runtime, and extend Ray beyond Python.

Cluster Deployment and Autoscaling

Ray's cluster lifecycle is governed by a small set of builder and runner entry points. ClientBuilder in python/ray/client_builder.py is the canonical client-side entry: it parses a connection target (a local address, a remote cluster address, or a cluster:// URI), validates it, and produces a RayContext used by ray.init to attach or start a cluster Source: python/ray/client_builder.py:1-40. For programmatic and test usage, python/ray/cluster_utils.py exposes Cluster and ClusterProc helpers that wrap the head/worker startup commands in a single Python object, making it trivial to spin up disposable clusters in unit tests and notebooks Source: python/ray/cluster_utils.py:1-80.

Node provisioning on cloud backends is implemented as pluggable *node providers* consumed by the autoscaler. The legacy StandardAutoscaler lives at python/ray/autoscaler/_private/autoscaler.py and coordinates the polling loop that matches desired capacity against observed cluster state, reconciling via the provider's create_node / terminate_node / non_terminated_nodes methods Source: python/ray/autoscaler/_private/autoscaler.py:1-60. The Kubernetes path is provided by KubeRayNodeProvider, which translates Ray's abstract node lifecycle into RayCluster custom-resource operations Source: python/ray/autoscaler/_private/kuberay/node_provider.py:1-80.

Ray 2.x introduces a redesigned autoscaler under python/ray/autoscaler/v2/. The AutoscalerV2 entry point schedules periodic reconcile passes and delegates per-instance state to InstanceManager, which owns a state machine over Instances (launching, running, stopping, stopped) and exposes try_schedule / request_instance for the higher-level scaler to call Source: python/ray/autoscaler/v2/autoscaler.py:1-80, Source: python/ray/autoscaler/v2/instance_manager/instance_manager.py:1-80. The split between a stateless controller (AutoscalerV2) and a stateful instance manager (InstanceManager) is the architectural pattern that makes the new autoscaler easier to test and reason about.

Runtime Observability

Observability in Ray is built on three legs: metrics, structured events on GCS pubsub, and log forwarding. Metrics are exported via PrometheusExporter hosted by a per-node MetricsAgent, which is started by the dashboard agent and scrapes registered Metric objects from core workers on a configurable interval Source: python/ray/_private/metrics_agent.py:1-80. The export endpoint, histogram bucket configuration, and agent registration are centralized as constants in python/ray/_private/ray_constants.py, including keys such as METRIC_CONFIG_REGISTER_BACKEND and METRIC_PUBLISH_PERIOD_MS that operators can tune via environment variables Source: python/ray/_private/ray_constants.py:1-120.

For event-style telemetry, python/ray/_private/gcs_pubsub.py implements a thin async wrapper around the GCS pubsub channel, allowing internal components (autoscaler, dashboard, job driver) to subscribe to job-, actor-, and node-lifecycle events without polling Redis. On the log side, LogMonitor reads per-worker stdout/stderr files written by the Raylet and forwards them to a centralized location, optionally indexed by the dashboard Source: python/ray/_private/log_monitor.py:1-60. Together these three channels are what the Ray Dashboard's "Cluster", "Jobs", "Actors", and "Metrics" views consume.

Cross-Language Extensibility

Although the user-facing API is Python-first, the Ray core is implemented in C++ and exposed to Python through Cython. The contract layer is declared in python/ray/includes/common.pxd, which exposes core types such as JobID, ActorID, TaskID, ObjectID, NodeID, and WorkerID as opaque structs backed by src/ray/common/id.h Source: python/ray/includes/common.pxd:1-60, Source: src/ray/common/id.h:1-60. These IDs are the lingua franca that every subsystem agrees on, which is what makes it possible for Python, C++, and Java clients to interoperate on the same cluster.

Function-level extensibility is mediated by RemoteFunction in python/ray/remote_function.py, which captures user-supplied metadata (resources, scheduling strategy, max retries, runtime env) and serializes it into the task spec proto delivered to the Raylet Source: python/ray/remote_function.py:1-80. The serialization path itself is in python/ray/_private/serialization.py, where pickle is wrapped in a registry-aware serializer that supports numpy zero-copy, PyArrow, and language-agnostic wire formats, with cross_language modes reserved for non-Python tasks Source: python/ray/_private/serialization.py:1-80. Worker bootstrap is centralized in python/ray/_private/worker.py, which initializes the mode (LOCAL / SPILL / CROSS_LANG), connects to the Raylet, and exposes the global worker singleton that all APIs call into Source: python/ray/_private/worker.py:1-80. The long-running Rust API request tracked in issue #20609 is exactly an effort to add another client on top of this same C++ core using the same ID types and serialization channel.

How the Subsystems Compose

Operators typically touch the first column, library authors the last, and application users only the first row. The bounded, layered design is what lets the Q3 2025 roadmap focus on reliability and DX without rewriting the core (Source: community issue #54923).

Doramagic Pitfall Log

Found 6 structured pitfall item(s), including 0 high/blocking item(s). Top priority: Capability evidence risk - Capability evidence risk requires verification.

1. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/ray-project/ray

2. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/ray-project/ray

3. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/ray-project/ray

4. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/ray-project/ray

5. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/ray-project/ray

6. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/ray-project/ray

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using ray with real data or production workflows.

• Ray-2.56.0 - github / github_release
• Ray-2.55.1 - github / github_release
• Ray-2.55.0 - github / github_release
• Ray-2.54.1 - github / github_release
• Ray-2.54.0 - github / github_release
• Ray-2.53.0 - github / github_release
• Ray-2.51.2 - github / github_release
• Ray-2.52.1 - github / github_release
• Ray-2.52.0 - github / github_release
• Ray-2.51.1 - github / github_release
• Capability evidence risk requires verification - GitHub / issue