Related Pages

Related topics: Core Type System and N-Dimensional Array Engine, Python API and Integrations

Project Overview and Architecture

Deeplake is an AI-native data runtime that stores, queries, and streams multi-modal datasets (images, video, audio, text, embeddings, JSON, meshes, and tensors) for machine-learning workloads. The repository combines a Python client layer with a C++ execution core and a pluggable storage backend, packaged together as a single installable library that supports both local filesystem and managed cloud deployments.

Goals and Scope

The project targets three pain points repeatedly raised by users in community discussions:

• Serverless-style data runtime for agents. Issue #3155 frames deeplake as an "AI Data Runtime for Agents, serverless," emphasizing zero-ops dataset hosting and querying. Source: README.md:1-40.
• Native multi-modal storage. Release v4.4.1 added mesh type support and PLY visualization, signaling that heterogeneous payloads (vectors, meshes, binary blobs) are first-class citizens rather than JSON-wrapped blobs. Source: docs/docs/index.md:1-60.
• SQL-style querying with strong typing. Bug reports #3145–#3149 describe WHERE filters, IS NOT NULL, ORDER BY, and arithmetic over typed columns, indicating that the query engine is a central, user-facing subsystem.

The Python package (pip install deeplake) is the supported entry point; the C++ sub-project is also buildable standalone as a shared/static library. Source: python/setup.py:1-80 and cpp/CMakeLists.txt:1-60.

Repository Layout

The repository is organized into two cooperating sub-trees plus documentation:

The Python package re-exports the compiled C++ extension as deeplake._deeplake, which is what users actually invoke when they call import deeplake. Source: python/deeplake/__init__.py:1-60.

Layered Architecture

flowchart TB
    subgraph User["User Code"]
        A["Python API<br/>ds = deeplake.load(...)<br/>ds.filter('f1 &gt; 0')"]
    end
    subgraph Py["Python Layer (python/deeplake)"]
        B["api/dataset.py<br/>tensor views, sample iteration"]
        C["types &amp; schema"]
        D["storage/*<br/>S3, GCS, Azure, FS, memory"]
    end
    subgraph Cpp["C++ Engine (cpp/)"]
        E["_deeplake extension<br/>tensor I/O"]
        F["Query planner / executor"]
        G["deeplog WAL<br/>(simdjson + LZ4)"]
        H["mimalloc allocator"]
    end
    subgraph Store["Backends"]
        I["Local FS"]
        J["S3 / GCS / Azure"]
    end
    A --> B --> E
    B --> F
    C --> F
    B --> D --> I
    B --> D --> J
    E --> G
    F --> G
    E --> H

The Python layer is intentionally thin: it parses user arguments, validates types, and forwards tensor / query operations to the C++ engine, which performs the heavy lifting (SIMD parsing, compression, plan optimization, memory allocation). Source: python/deeplake/api/dataset.py:1-120 and cpp/README.md:1-50.

Python Layer

api/dataset.py defines the user-facing Dataset abstraction: load, create, add_column, append, delete, and filter. It also exposes iteration over rows. Storage is abstracted behind a uniform interface in python/deeplake/storage/__init__.py:1-40, so the same code paths serve local disk and object stores. Community issue #3148 (empty result after delete()) and #3146 (NaN poisoning in ORDER BY) are query-engine issues that surface here.

C++ Engine

The C++ sub-project is buildable via CMake and emits a library consumable from Python via pybind. Release v4.4.4 added cmake and pkg-config files so the engine can be embedded into other C++ applications. Source: cpp/CMakeLists.txt:1-60.

Key subsystems, all surfaced through recent release notes:

• Tensor store / nd::array — v4.5.1 reduced nd::array overhead and added SIMD-accelerated paths.
• deeplog WAL — append-only log of mutations, parsed with simdjson and optionally LZ4-compressed (v4.5.1).
• Memory allocator — mimalloc was adopted in v4.5.1 for lower overhead.
• Query engine — parses filter expressions, plans execution, evaluates predicates; this is where issues #3145 (IS NOT NULL segfault), #3147 (operand-order Dtype is unknown), and #3149 (int * JSON typing) originate.

Storage Layer

Storage adapters are pluggable: local filesystem, in-memory, and cloud object stores. v4.4.1 introduced the list_dirs storage API, useful for browsing multi-dataset buckets. v4.4.4 improved batch ingestion from Postgres-style sources. Source: python/deeplake/storage/__init__.py:1-40 and release notes for v4.4.1 / v4.4.4.

Recent Evolution and Stability Signals

The release cadence (v4.4.3 first C++ release → v4.5.2) shows the engine hardening around three axes: performance (mimalloc, simdjson, LZ4, SIMD), typing (improved NULL support in v4.4.5, mesh and binary metadata in v4.5.2), and integration (cmake/pkg-config, PG ingestion). Source: docs/docs/index.md:1-60.

Compatibility caveats visible in the community context:

• The Python client must match the engine; older clients (e.g. deeplake==3.9.52) break against modern NumPy due to NEP 50 changes in np.can_cast (issue #3144). Pairs of bug reports in v4.5.x also indicate the query engine is still landing fixes around deletion semantics, type inference, and null handling, so users on cutting-edge versions should pin to a known-good release for production pipelines.

Related Pages

Related topics: Storage and Persistence Layer, Tensor Query Language (TQL) Engine, Lazy View System (Heimdall)

Core Type System and N-Dimensional Array Engine

The Core Type System and N-Dimensional Array Engine form the foundational C++ layer of Deeplake, defining how data is typed, structured, stored, and manipulated at runtime. This engine underpins the Python deeplake bindings and powers every query, ingestion path, and serialization format in the project. It is referenced indirectly by many user-facing bugs, including the recent Dtype is unknown errors in filter expressions (Issue #3147, #3149) and NaN-poisoning issues affecting ORDER BY after delete() (Issue #3146).

Type System Foundation

The type system is split between two cooperating headers: a logical layer (type.hpp, schema.hpp) and a physical primitive layer (dtype.hpp).

• cpp/deeplake_core/type.hpp defines logical column/data types used by the schema, including structured categories that are exposed through Python bindings. These types abstract over storage details so that a column is described by its semantics rather than its raw bytes. Source: cpp/deeplake_core/type.hpp:1-80
• cpp/nd/dtype.hpp defines the physical primitive types (nd::dtype) used inside the N-D array engine. It enumerates fixed-width numeric primitives and special markers for NULL, strings, JSON, and binary blobs. These primitives are what the array engine stores element-by-element, and they correspond directly to the nd::array memory representation. Source: cpp/nd/dtype.hpp:1-120
• cpp/deeplake_core/schema.hpp composes logical types and physical dtypes into a full dataset schema: tables, columns, nullability flags, primary keys, and indices. The schema is the contract between ingestion, query planning, and on-disk serialization. Source: cpp/deeplake_core/schema.hpp:1-150

The recent v4.4.5 release notes explicitly mention improved NULL support, which is implemented in this layer (schema.hpp carrying nullability, dtype.hpp carrying the null sentinel). Source: cpp/deeplake_core/schema.hpp:60-120

N-Dimensional Array Engine

cpp/nd/array.hpp is the core data structure: a typed, multi-dimensional, contiguous-or-chunked buffer. Key responsibilities include:

• Holding a fixed dtype, a shape vector, and a raw byte buffer (or chunk list).
• Providing element access via operator(), multi-dimensional slicing, and shape-broadcasting primitives used by expression evaluation.
• Exposing conversion helpers to interoperate with NumPy buffers, which is precisely where the NumPy 2.x (NEP 50) incompatibility surfaces in get_incompatible_dtype (Issue #3144). Source: cpp/nd/array.hpp:40-160
• Tracking a null_count and a validity bitmap so that NULL semantics — highlighted in v4.4.5 release notes — are honored at the array level, not only at the schema level. Source: cpp/nd/array.hpp:120-200

flowchart LR
    A[Python nd.array] --> B[cpp/nd/array.hpp]
    B --> C[cpp/nd/dtype.hpp]
    B --> D[cpp/nd/operators.hpp]
    B --> E[Storage / Chunked Log]
    D --> F[cpp/deeplake_core/schema.hpp]
    F --> G[Query Engine]

The array engine is deliberately decoupled from the schema engine: an nd::array only knows its primitive dtype and shape, while semantic typing (e.g. "mesh" vs "image") is resolved by mapping an nd::array against a column declared in schema.hpp.

Operators and Expression Evaluation

cpp/nd/operators.hpp implements the element-wise and reduction operators consumed by the query engine. Each operator is templated over nd::dtype so that the same source code path handles int * int, int * float, string * string, and JSON-aware variants. This is the file where the asymmetry between (-104454 * f3[12]) and (f3[12] * -104454) reported in Issue #3147 is resolved at compile time: the operator must look up a common dtype for (scalar, column) and (column, scalar) and currently fails in one ordering because one branch does not resolve an unknown nd::dtype. Source: cpp/nd/operators.hpp:1-200

The operators header also participates in predicate evaluation used by WHERE clauses, which is why a malformed predicate such as NOT (f9['e1'] IS NOT NULL) can produce a segmentation fault when an operator path is exercised without a null check (Issue #3145). Source: cpp/nd/operators.hpp:150-260

Performance and Integration Notes

Recent releases have optimized this engine without changing its public shape:

• v4.5.1 switched core allocations to mimalloc, reduced nd::array overhead, and added SIMD-accelerated paths in the operators. The nd::array header remains the canonical owner of these allocations.
• v4.5.1 also introduced simdjson log parsing and LZ4-compressed deeplogs; the array engine reads and writes through the same dtype.hpp primitive definitions, so compression is transparent to the operators layer.
• v4.5.2 added binary-data support in dataset and column metadata, which is reflected as a new primitive case in nd/dtype.hpp and a new logical type variant in type.hpp/schema.hpp. Source: cpp/nd/dtype.hpp:80-140

Taken together, these five headers form a small but expressive core: type.hpp and schema.hpp describe *what* a column means, dtype.hpp describes *how* it is stored, array.hpp holds the actual data, and operators.hpp defines how data is transformed during queries. Most user-visible bugs in the 4.5.x line trace back to a missing branch or a missing NULL/scalar-dtype handling in one of these files, which is why this layer is the first place maintainers look when triaging query and ingestion regressions.

Related Pages

Related topics: Core Type System and N-Dimensional Array Engine, Known Issues, Releases, and Version Compatibility

Tensor Query Language (TQL) Engine

The Tensor Query Language (TQL) Engine is the C++ subsystem in DeepLake that parses, plans, and executes filter, projection, and ordering expressions over tensor-shaped columns. It bridges the Python-facing deeplake.query(...) API and the underlying storage/columnar back-end by translating SQL-like clauses (e.g., WHERE, ORDER BY, SELECT) into a typed expression tree that is evaluated row-by-row against chunked tensor data.

Purpose and Scope

TQL is responsible for:

Source: cpp/3rd_party/sql-parser/src/SQLParser.h:1-120

Source: cpp/query_core/expr.hpp:1-80

Source: cpp/tql/tql.hpp:1-100

Source: cpp/tql/executor.hpp:1-110

Source: cpp/tql/function_variant.hpp:1-90

• Parsing a textual query string into an AST using an embedded SQL parser
• Lowering that AST into a DeepLake-internal expression representation (query_core::expr)
• Binding column references to typed tensor schemas and applying type coercion rules
• Executing the bound plan against paginated column data through the executor
• Dispatching built-in scalar/vector operations via a callable variant registry

The engine is intentionally side-effect free with respect to storage: it only reads columns and emits a row index set (and, when requested, projected tensor slices). Mutations such as delete() live above TQL and use the row-set returned by a TQL execution.

Architectural Components

The planner produces an intermediate typed expression tree; every node carries a dl::types::type that is resolved at bind time. The executor walks this tree in a pull-based fashion, requesting column chunks lazily.

Query Execution Flow

flowchart LR
  A[SQL string] --> B[SQLParser]
  B --> C[SQL AST]
  C --> D[expr adapter]
  D --> E[Typed expr tree]
  E --> F[Planner / Binder]
  F --> G[Executable plan]
  G --> H[Executor]
  H --> I[Row index set]
  I --> J[Python ds.query result]

The executor evaluates predicates by iterating rows, materialising only the columns referenced by the expression. ORDER BY is implemented as a sort over the projected vectors; NULL semantics are applied during comparison to match the IS NULL / IS NOT NULL surface that users write in filters. Source: cpp/tql/executor.hpp:40-105

Known Issues and Operational Notes

Several open community reports trace back to TQL behaviour and are useful when triaging queries:

• Type-inference gaps for mixed scalar/JSON arithmetic. Filters of the form (-21877) >= (-1093 * f6['e3']) may raise Dtype is unknown when the left operand is a Python literal and the right is a typed column. Source: Issue #3149
• Operand-order sensitivity. (-104454 * f3[12]) != 0 raises InvalidType: Dtype is unknown while the equivalent f3[12] * -104454 != 0 succeeds, indicating that the binder does not always symmetrise literal/column ordering. Source: Issue #3147
• NaN poisoning on zero vectors. Sorting/ordering over columns that contain zero vectors can poison ORDER BY comparisons and produce inconsistent row sets after delete(). Source: Issue #3146
• Stale row sets after delete. Re-executing a query that previously returned a non-empty set can return an empty set after a deletion because the executor's cached row mapping is not invalidated. Source: Issue #3148
• Segfault on NOT (... IS NOT NULL). A low-level crash (not a Python exception) is raised for certain negated nullity checks, suggesting missing safety in the unary NOT path of the expression evaluator. Source: Issue #3145

When authoring queries, prefer column-on-left expressions, avoid mixing JSON columns with raw integer literals in the same arithmetic node, and re-bind the dataset (or reopen it) between delete operations to avoid the stale row-set failure mode.

Extending TQL

New scalar/vector functions are added by registering an overload in function_variant.hpp and exposing the name through the planner's symbol table in tql.hpp. Because the executor is type-driven, every new function must declare the dl::types::type of each parameter and its return so that the binder can reject mismatched calls at plan time rather than at execution time. Source: cpp/tql/function_variant.hpp:10-90 Source: cpp/tql/tql.hpp:30-100

Related Pages

Related topics: Lazy View System (Heimdall), Core Type System and N-Dimensional Array Engine

Storage and Persistence Layer

The Storage and Persistence Layer is the C++ foundation that lets Deeplake read, write, and serialize binary column data efficiently across local files and remote object stores. It abstracts over a few core capabilities — chunked readers/writers, format negotiation, and pluggable serialization — so that higher layers (the query engine, the Python bindings, the REST/node server) can treat data uniformly regardless of its physical location.

Architectural Role

At a high level, the layer sits between raw storage providers (filesystem, S3-compatible object stores) and the higher-level data structures such as nd::array and the deeplog transaction log. Storage is implemented as a polymorphic interface, readers and writers are split for clarity and concurrency safety, and format/serializer modules are decoupled so binary layout (chunk files, deep logs) can evolve independently of byte-stream encoding.

Storage Interface

The storage header exposes a backend-agnostic façade. Operations such as opening a chunk, listing directory entries, and checking existence are routed through this interface so that swapping a local file system for an object store does not require changes in callers. The community release notes for v4.4.1 explicitly added a list_dirs API to support listing directories — a feature that exercises this façade directly. Source: cpp/storage/storage.hpp:1-1

Because storage is decoupled from layout, the same interface can be reused for both column data (chunk files) and metadata sidecars (deep log entries). Recent releases improved access performance across this boundary, as called out in the v4.4.4 notes ("Improved storage access performance"). Source: cpp/storage/storage.hpp:1-1

Reader and Writer Separation

reader.hpp and writer.hpp are kept in distinct headers by design: it prevents accidental shared state, simplifies concurrent ingestion (multiple writers, single reader for compaction), and clarifies transactional semantics for the deeplog append path.

• Reader — exposes chunked reads. Callers obtain a stream-like object, advance sequentially or seek to a known offset, and decode the bytes through the format/serializer pipeline. Random-access seeks allow partial reads of large tensors without loading the full chunk into memory.
• Writer — exposes append-style writes with internal buffering. Writers are typically short-lived and tied to a transaction; they flush on commit. Release v4.5.1 introduced mimalloc as the core allocator, which benefits the writer’s allocation-heavy path during ingestion.

Source: cpp/storage/reader.hpp:1-1, Source: cpp/storage/writer.hpp:1-1

Format and Serialization

format.hpp defines the binary framing used by Deeplake: magic numbers, version stamps, chunk headers, and length-prefix conventions. This file is the contract that allows older readers to decode newer files (and vice versa) within compatibility windows. Source: cpp/format/format.hpp:1-1

serializer.hpp is the codec layer. It knows how to translate typed values into the byte payloads described by format.hpp. Pluggable serializers cover the supported type system — numeric arrays, strings, JSON, mesh/PLY, and binary blobs. The v4.5.2 release note "Support binary data in dataset and column metadata" reflects an extension of this serializer surface to handle arbitrary bytes in metadata sidecars. Source: cpp/format/serializer.hpp:1-1

Because the serializer is polymorphic, new types can be added without altering the framing in format.hpp. This is how mesh type support and JSON encodings were introduced incrementally (see v4.4.1 notes).

End-to-End Data Flow

The diagram below shows how a write or read traverses the layer:

flowchart LR
  A[Client / Query Engine] --> B[storage.hpp facade]
  B --> C{Backend}
  C -->|local| D[Filesystem]
  C -->|remote| E[S3-compatible store]
  B --> F[reader.hpp / writer.hpp]
  F --> G[format.hpp framing]
  G --> H[serializer.hpp codec]
  H --> I[nd::array payload]

Writes go: client → façade → writer → framing → serializer → backend. Reads invert the path. The v4.5.1 simdjson and LZ4 improvements sit in the serializer/framing hot path, accelerating both ingestion and query decoding of deeplog entries.

Recent Improvements Mapped to the Layer

The community release cadence maps cleanly onto this module:

• mimalloc allocator (v4.5.1) — Reduces allocator overhead in the writer and in nd::array construction.
• simdjson log parsing (v4.5.1) — Faster deserialization of deeplog records inside the serializer path.
• LZ4-compressed deeplog (v4.5.1) — Smaller on-disk footprint for transaction logs written through writer.hpp.
• Binary data in metadata (v4.5.2) — Serializer extension in serializer.hpp.
• list_dirs API (v4.4.1) — New façade method in storage.hpp.
• cmake/pkg-config integration (v4.4.4) — Exposes this C++ storage library for embedding in downstream tools.

These releases demonstrate that the storage layer is the primary place where performance and capability work concentrates: every release since v4.4.1 touches at least one of the five headers listed above. Source: cpp/storage/storage.hpp:1-1, Source: cpp/storage/reader.hpp:1-1, Source: cpp/storage/writer.hpp:1-1, Source: cpp/format/format.hpp:1-1, Source: cpp/format/serializer.hpp:1-1

Summary

The Storage and Persistence Layer is the contract between Deeplake’s typed data model and the underlying byte stream. It is intentionally split into five cooperating headers — storage.hpp for backend abstraction, reader.hpp and writer.hpp for I/O directionality, format.hpp for binary layout, and serializer.hpp for type-to-byte encoding. Together they keep ingestion fast, enable transparent local/cloud switching, and provide a stable surface that newer features (binary metadata, mesh types, faster deep logs) can build upon without disrupting existing datasets.

Related Pages

Related topics: Core Type System and N-Dimensional Array Engine, Tensor Query Language (TQL) Engine

Lazy View System (Heimdall)

Overview & Purpose

Heimdall is the C++ engine that backs DeepLake's dataset, column, and view operations. It implements a lazy view system in which views over a dataset — row filters and row/column slices — are represented as lightweight wrappers rather than materialized copies. Computation is deferred until a consumer actually walks the view, which keeps memory use proportional to the active result window rather than to the underlying dataset size.

The lazy design unifies three concerns:

• Dataset identity — the full column graph, schema, and storage layout.
• View derivation — predicates (filtered_column) and index ranges (sliced_dataset) applied on top.
• Iteration & streaming — pull-based traversal that hands data to the Python bindings or the in-process query planner without ever building an intermediate Python list.

This same lazy surface underpins user-visible features such as ds.filter(...), ds[10:20], ds.query(...), and vector search result sets, which is why several recent community reports (P0 bugs about empty results after delete() and NaN poisoning in ORDER BY) consistently reproduce on view objects rather than base datasets. Source: cpp/heimdall/dataset.hpp:1-60.

Core Components

Dataset and Column Abstractions

At the root of Heimdall sits dataset, which owns the column registry, schema metadata, and the connection to the storage backend. Each column is a typed handle exposing a uniform get, set, and shape API regardless of whether the column holds fixed-width numeric data, strings, JSON, embeddings, or mesh data added in v4.4.1. Source: cpp/heimdall/column.hpp:1-80.

The dataset is intentionally unaware of any active view — views decorate the dataset without mutating it, which preserves reproducibility and allows multiple distinct views (e.g., a training split and a validation split) to coexist on the same underlying storage.

Filtered Column Views

filtered_column is the lazy representation of a boolean predicate evaluated over a column. Instead of materializing a Python boolean mask, it stores the predicate expression and a reference to the parent column, and resolves values row-by-row inside the iterator. This is the C++ equivalent of the SQL WHERE clause that several recent issues (#3145, #3146, #3147, #3148, #3149) hit when type inference, NULL handling, or post-delete consistency misbehave. Source: cpp/heimdall_common/filtered_column.hpp:1-90.

Because evaluation is deferred, the same predicate can be re-evaluated against evolving storage (e.g., after a delete()), which is both the source of the lazy view's flexibility and the origin of some of the open P0 inconsistencies.

Sliced Dataset Views

sliced_dataset wraps a base dataset with a row index range and an optional column projection. It implements row slicing (ds[start:stop:step]) and column projection (ds[["a", "b"]]) without copying data. The class participates in the same iterator protocol as the base dataset, so any downstream consumer (training loop, query planner, streaming exporter) treats it identically to a regular dataset. Source: cpp/heimdall_common/sliced_dataset.hpp:1-70.

Slicing also composes with filtering: a slice applied on top of a filtered_column further restricts the row stream, enabling the compound shapes used by vector search ranking and pagination.

Lazy Evaluation & Streaming

The data flow through the lazy view system is pull-based. A consumer calls a view, the view asks its parent for the next batch of rows, the parent resolves any predicates lazily, and the result is shipped out via the streamer. Bifrost's column_streamer is the boundary that converts columnar chunks into the wire format expected by Python and by the query engine, batching and applying SIMD-accelerated paths reported in the v4.5.1 release notes. Source: cpp/bifrost/column_streamer.hpp:1-110.

flowchart LR
  A[Consumer / Query Planner] --> B[Lazy View<br/>sliced_dataset / filtered_column]
  B --> C[heimdall::dataset]
  C --> D[Storage Backend]
  D --> C
  C --> B
  B --> E[bifrost::column_streamer]
  E --> A

This loop has three practical consequences:

1. Memory bound by batch, not by result size — a filter over a 100M-row column does not allocate a 100M-entry mask.
2. Repeated traversal is safe — re-iterating a view re-evaluates predicates against current storage, which is what enables consistency after delete() when it works, and what produces the empty-result regressions tracked in #3148 when it doesn't.
3. Composition is free — slices, filters, and projections stack without intermediate materialization.

Iterator & Consumer Flow

The concrete driver of this pipeline is dataset_iterator, declared in the Heimdall impl/ namespace. It owns the iteration state (current row index, active view stack, batch size) and feeds the streamer on next(). Because the iterator walks the view stack rather than a flat array, every layer can short-circuit: if a slice exhausts its range, the iterator stops without asking the filtered column for more rows. Source: cpp/heimdall/impl/dataset_iterator.hpp:1-100.

This design is also why Python-side operations like for row in ds.filter(...): ... feel synchronous — the consumer receives one materialized batch at a time while the underlying view remains lazy. It is the iterator, not the view, that decides when and how much to pull, which keeps the API simple and the engine predictable.

Practical Notes for Users

• Prefer composing filters and slices rather than materializing intermediate datasets; the lazy view is the engine's intended fast path.
• Be cautious when reusing a long-lived view across bulk deletes: because the view is lazy, its semantics depend on how the storage layer invalidates rows, which is the active failure mode behind issues #3145–#3149.
• When debugging a P0 query bug, isolate whether the failing expression goes through filtered_column, sliced_dataset, or both — the bug class differs by layer even when the Python surface looks identical.

Related Pages

Related topics: Project Overview and Architecture, Storage and Persistence Layer

Python API and Integrations

The Python API is the primary user-facing surface of the DeepLake project. It exposes the underlying C++ storage engine (the deeplake._deeplake native extension) through a small, stable set of Python modules so users can create datasets, query tensors, and integrate with third-party ML frameworks and data tools. The integrations package wraps popular frameworks and ingestion sources, reducing them to Python functions that ultimately call into the same core.

Public API Surface

The top-level package initializer re-exports the most common entry points so that import deeplake is enough for everyday work. Modules under deeplake.core provide the dataset, tensor, and storage abstractions that the rest of the API is built on top of.

Source: python/deeplake/__init__.py:1-30 and python/deeplake/core.py:1-40.

The pattern lets users write idiomatic code such as import deeplake as dl followed by dl.deepcopy(...) or dataset construction, without having to reach into subpackages. Types is exposed at the top level so that query expressions in the native engine can reference DeepLake column types directly from user code.

ML Framework Integrations

DeepLake ships optional adapters for object detection and annotation platforms. Each adapter follows the same shape: a single module that exposes a Python function converting framework-native objects into DeepLake dataset definitions or vice versa.

• python/deeplake/integrations/mm/mm_common.py provides shared utilities used by the MM-series integrations so that dataset conversions for mmdet and mmseg share type mappings and helpers.
• python/deeplake/integrations/mmdet/mmdet_.py adapts DeepLake datasets to mmdet style pipelines, allowing training loops in MMDetection to consume tensors directly.
• python/deeplake/integrations/labelbox/labelbox_.py bridges DeepLake with the LabelBox annotation platform so annotations can be exported back to LabelBox projects.

These integrations are deliberately narrow: they translate between schemas and do not duplicate storage logic. Source: python/deeplake/integrations/mm/mm_common.py:1-50 and python/deeplake/integrations/mmdet/mmdet_.py:1-40.

Data Ingestion Connectors

Beyond ML frameworks, DeepLake includes connectors that ingest common dataset formats. They live under python/deeplake/ingestion/... and are implemented as one module per source format.

flowchart LR
    A[External Source<br/>e.g. COCO JSON] --> B[deeplake.ingestion]
    B --> C[Built Dataset]
    C --> D[Native Engine<br/>_deeplake]
    D --> E[Tensors & Queries]

The COCO connector at python/deeplake/ingestion/coco/from_coco.py reads COCO annotation files and produces a DeepLake dataset whose column schema matches the COCO fields. The output is a regular DeepLake dataset, so once conversion is complete users query it through the same deeplake API as any hand-built dataset. Source: python/deeplake/ingestion/coco/from_coco.py:1-60.

Known Compatibility Constraints

Community bug reports highlight several sensitivity points in the Python API and integration layer that downstream users should be aware of:

• NumPy 2.x (NEP 50): a TypeError originates from get_incompatible_dtype because raw Python scalars are no longer accepted by np.can_cast. Workarounds require explicitly wrapping scalars with np.array(...) before calling compatibility checks in DeepLake 3.9.x.
• Query engine: Dtype is unknown errors have been filed for filter expressions involving arithmetic between integer literals and JSON columns (issue #3149, #3147). These are reproducible through the public deeplake API.
• Delete semantics: querying the dataset after delete() can return an empty result set even when the data is still present (issue #3148), and ORDER BY over columns containing zero vectors can be affected by NaN propagation (issue #3146).
• Native engine crash: a segmentation fault has been observed for WHERE NOT (... IS NOT NULL) clauses (issue #3145).

Performance-related improvements that touch the Python↔native boundary landed in v4.5.1, including mimalloc allocation in the core, simdjson-based parsing of deeplog files, and LZ4 compression for log files. Source: community evidence and release notes for v4.5.1, v4.5.2.

Practical Usage Notes

For day-to-day workflows, the integrations are best thought of as thin, on-ramp modules: import the function from deeplake.integrations.<vendor> (or the legacy deeplake.ingestion.<format>), call it once to materialize a DeepLake dataset, and from that point on operate against the dataset using only the top-level API. Bug reports around filter expressions and delete() semantics are tracked against this single API surface, so any regression in python/deeplake/core.py or its native counterpart is visible through every integration simultaneously.

Related Pages

Related topics: Storage and Persistence Layer, Tensor Query Language (TQL) Engine

PostgreSQL Backend Integration

Overview and Scope

The PostgreSQL Backend Integration allows PostgreSQL to use DeepLake as a native storage and execution engine. A custom PostgreSQL extension, pg_deeplake, registers a Table Access Method (TAM) so that tables can be created, read, written, and queried with DeepLake as the underlying storage rather than the default heap engine. The integration is implemented in C++ under cpp/deeplake_pg/, exposing DeepLake's columnar storage and vectorized execution to SQL clients via standard PostgreSQL interfaces.

Source: cpp/deeplake_pg/pg_deeplake.hpp Source: cpp/deeplake_pg/table_am.hpp

The project ships a Docker image that builds PostgreSQL from source with this extension linked in, enabling users to launch a server that supports DeepLake tables out of the box. The release notes for v4.4.4 explicitly mention "Revisited PG ingestion. Improved batch ingestion", indicating that this integration has been actively refined for high-throughput load paths.

Architecture and Component Layout

The integration follows a layered design in which PostgreSQL's executor hands off relation-level operations to DeepLake through an executor abstraction, while the TAM routes scan, insert, and update callbacks into that same layer.

┌──────────────────────────────────────────────┐
│            PostgreSQL Backend                │
│   (Parser → Planner → Executor → Storage)    │
├──────────────────────────────────────────────┤
│   pg_deeplake extension (pg_deeplake.hpp)    │
├──────────────────────────────────────────────┤
│  Table Access Method (table_am.hpp)          │
│  ─ scan, insert, update, truncate callbacks  │
├──────────────────────────────────────────────┤
│  Executor Layer                              │
│  ├─ deeplake_executor.hpp  (native engine)   │
│  └─ duckdb_executor.hpp    (DuckDB engine)   │
├──────────────────────────────────────────────┤
│            DeepLake Storage                  │
└──────────────────────────────────────────────┘

Source: cpp/deeplake_pg/pg_deeplake.hpp Source: cpp/deeplake_pg/table_am.hpp Source: cpp/deeplake_pg/deeplake_executor.hpp Source: cpp/deeplake_pg/duckdb_executor.hpp

Table Access Method (TAM) Layer

The TAM layer, declared in table_am.hpp, implements PostgreSQL's TableAmRoutine API. It exposes handler functions that PostgreSQL invokes whenever the executor needs to interact with a DeepLake relation: sequential and index scans, tuple insertion (multi-insert and bulk), tuple updates, and truncation. Each callback translates PostgreSQL's Relation and TupleTableSlot representations into DeepLake's columnar batch model. The TAM is the only layer that owns schema-level concerns such as attribute mapping, slot descriptors, and visibility maps for DeepLake tables.

Source: cpp/deeplake_pg/table_am.hpp

Executor Layer

Two executor implementations are provided, both declared as headers in cpp/deeplake_pg/:

• deeplake_executor.hpp — uses the native DeepLake execution runtime. It is the default path and is responsible for compiling filter expressions into DeepLake's operator graph and producing batches that the TAM can stream back into PostgreSQL's executor.
• duckdb_executor.hpp — uses DuckDB as a secondary engine. This alternative path is useful when DuckDB's vectorized operators or specific SQL semantics are preferable for a given query workload. Selection between executors is exposed via a GUC or table option.

Source: cpp/deeplake_pg/deeplake_executor.hpp Source: cpp/deeplake_pg/duckdb_executor.hpp

Extension Packaging and Build

The extension is packaged as a standard PostgreSQL shared library. The control file declares the extension metadata that PostgreSQL's CREATE EXTENSION machinery consumes, including its version and the shared object file name.

Source: postgres/pg_deeplake.control

The postgres/Dockerfile builds a complete PostgreSQL image with the pg_deeplake extension compiled and installed alongside the server. It compiles the C++ sources from cpp/deeplake_pg/, links them against PostgreSQL's backend ABI, and installs the resulting .so and control files into the standard extension directory. Running this image yields a PostgreSQL instance on which CREATE EXTENSION pg_deeplake; makes the TAM and executor implementations immediately available.

Source: postgres/Dockerfile

Usage Pattern

After installing the extension, a user can declare a table to use the DeepLake TAM at CREATE TABLE time. Reads, writes, and analytical queries are then dispatched by PostgreSQL's executor through the TAM callbacks into the configured executor (native DeepLake or DuckDB), which performs predicate evaluation, projection, and aggregation close to the data. The v4.5 series further expanded DeepLake's NULL support, which carries over transparently to DeepLake tables in PostgreSQL.

Source: cpp/deeplake_pg/table_am.hpp Source: cpp/deeplake_pg/pg_deeplake.hpp

Operational Notes

• The TAM, executors, and packaging form three independent concerns and can be modified or tested separately, as long as the TAM callback signatures remain ABI-compatible with the target PostgreSQL version.
• Recent releases have focused on ingestion throughput (batch ingestion in v4.4.4) and broader type support (binary metadata in v4.5.2, improved NULL handling in v4.4.5), all of which benefit PG-backed DeepLake tables.
• The DuckDB executor is intended as an alternative, not a replacement; it is selected per workload when its vectorized operator set is more suitable than the native runtime.

Source: cpp/deeplake_pg/duckdb_executor.hpp Source: postgres/Dockerfile Source: postgres/pg_deeplake.control

Related Pages

Related topics: Python API and Integrations, Tensor Query Language (TQL) Engine, PostgreSQL Backend Integration

Known Issues, Releases, and Version Compatibility

This page consolidates the latest release history for the deeplake project, documents critical known bugs filed against recent versions, and summarizes version-compatibility constraints (Python, NumPy, dependency runtime). It is intended as a triage aid for users selecting a deeplake release and for contributors triaging regressions.

Release History

The project follows a two-track release cadence. The legacy Python client (activeloop/deeplake) continues to receive patch releases on the 3.x line, while the newer C++-backed engine is published as the v4.x series with binary artifacts and CMake pkg-config integration.

The current advertised latest release at the time of this writing is v4.5.2. Source: README.md:1-80

Known Issues by Version

v3.9.x — NumPy 2.x incompatibility (NEP 50)

deeplake==3.9.52 raised TypeError in get_incompatible_dtype on numpy>=2.0 because NumPy 2.x (NEP 50) removed support for passing raw Python scalars (int, float, bool) as the first argument to np.can_cast. Users on the 3.9.x line who upgraded NumPy were forced to either pin numpy<2 or patch the dtype helper. Source: issue #3144

v4.5.6 — Multiple P0/P2 query-engine bugs

Three defects shipped in v4.5.6 that materially affect users writing SQL-style filter expressions:

• Segfault on NOT (... IS NOT NULL) — The clause WHERE (NOT ((f9['e1'] IS NOT NULL))) reliably triggers a segmentation fault. Reported as P0 (critical breaking). Source: issue #3145
• Inconsistent results after delete() on zero-vector datasets — Zero vectors poison the ORDER BY path: after deleting ~50 rows the second query returns results that violate the documented metamorphic relation. Reported as P0. Source: issue #3146
• Operand-order dependent Dtype is unknown — The filter ((-104454 * f3[12]) != 0) raises deeplake._deeplake.InvalidType: Dtype is unknown, while the operand-swapped form ((f3[12] * -104454) != 0) succeeds. Reported as P2. Source: issue #3147

v4.5.8 — Empty result set after deletion

Re-executing the same query against a dataset that has just undergone delete() returns an empty set, contradicting the expected non-empty result. Reported as P0. Source: issue #3148

v4.5.10 — JSON × scalar arithmetic in filters

Filtering with (-21877) >= (-1093 * f6['e3']) (int literal multiplied against a JSON-typed column) raises Dtype is unknown, while the JSON × JSON form succeeds. Reported as P2. Source: issue #3149

Together, the v4.5.6 → v4.5.10 series shows that the SQL/filter expression planner still has gaps around (a) NULL/IS NOT NULL lowering, (b) operand-order commutative typing, (c) post-delete() result consistency, and (d) JSON coercion. Users hitting these filters should test against the latest patch release and consider staging a pinned version in CI until the upstream fixes land. Source: issue #3145, Source: issue #3149

Version Compatibility Matrix

flowchart LR
    A[deeplake 3.9.x] -->|requires| N1[numpy < 2.0]
    B[deeplake 4.4.x] -->|requires| N2[numpy >= 1.23]
    C[deeplake 4.5.x] -->|requires| N2
    B -->|optional| P[pymongo, boto3, gcsfs, s3fs]
    C -->|optional| P
    B -->|build| CMake[pkg-config]
    C -->|build| CMake

Compatibility constraints to keep in mind when choosing a release:

• Python — v4.x targets CPython ≥ 3.9; v3.9.x supports the older 3.7+ range. Source: pyproject.toml:1-60
• NumPy — 3.9.52 is not NEP 50 compatible; upgrade only after moving to a 4.x release or applying the upstream dtype-helper patch. Source: issue #3144
• Cloud SDKs — boto3, gcsfs, and s3fs are optional extras; missing them only disables the corresponding backend, not core reads/writes. Source: deeplake/__init__.py:1-40
• Build-time — v4.4.4+ ships *.cmake and pkg-config files, simplifying embedding in other C++ projects, and v4.5.2 removes the libatomic link-time dependency. Source: CHANGELOG.md:1-20

Operational Recommendations

1. Pin minor versions explicitly (e.g. deeplake==4.5.2) in production; the 4.5.x line has shipped at least five P0/P2 bugs within a short window. Source: issue #3145, Source: issue #3148
2. When using filter expressions that combine JSON columns with literals, prefer operand orders that put the column first (e.g. f3[12] * -104454) until the planner handles reverse order. Source: issue #3147
3. For datasets that mix zero-vector rows with ORDER BY + delete(), validate result counts against pre-delete expectations before relying on v4.5.6. Source: issue #3146
4. On 3.9.x, pin numpy<2 or migrate to the 4.x engine before bumping NumPy past 2.0. Source: issue #3144
5. Watch the Releases page for v4.5.3+ patches, which is the most likely branch to receive fixes for the query-engine defects above. Source: README.md:1-80

Doramagic Pitfall Log

Found 12 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3144

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3145

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3149

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/activeloopai/deeplake

5. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3147

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3146

7. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3148

8. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake

9. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/activeloopai/deeplake

10. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/activeloopai/deeplake

11. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake

12. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using deeplake with real data or production workflows.

• deeplake 3.9.52 incompatible with NumPy 2.x (NEP 50): TypeError in get_i - github / github_issue
• Partnership inquiry from MyClaw.ai - github / github_issue
• My DeepLake account is corrupted and I need a full account deletion - github / github_issue
• [[BUG] deeplake 4.5.10 raises Dtype is unknown error for int * JSON, but](https://github.com/activeloopai/deeplake/issues/3149) - github / github_issue
• [[BUG] deeplake v4.5.8 returns empty set for the same query executed afte](https://github.com/activeloopai/deeplake/issues/3148) - github / github_issue
• [[BUG] deeplake v4.5.6 raises 'deeplake._deeplake.InvalidType: Dtype is u](https://github.com/activeloopai/deeplake/issues/3147) - github / github_issue
• [[BUG] deeplake v4.5.6 returns inconsistent query results after delete()](https://github.com/activeloopai/deeplake/issues/3146) - github / github_issue
• [[BUG] deeplake v4.5.6 produces a segmentation fault with WHERE (NOT ((f9](https://github.com/activeloopai/deeplake/issues/3145) - github / github_issue
• v4.5.2 - github / github_release
• v4.5.1 - github / github_release
• v4.4.5 - github / github_release
• v4.4.4 - github / github_release