# https://github.com/activeloopai/deeplake Project Manual Generated at: 2026-07-20 21:48:56 UTC ## Table of Contents - [Project Overview and Architecture](#page-overview) - [Core Type System and N-Dimensional Array Engine](#page-type-system) - [Tensor Query Language (TQL) Engine](#page-tql-engine) - [Storage and Persistence Layer](#page-storage-layer) - [Lazy View System (Heimdall)](#page-lazy-views) - [Python API and Integrations](#page-python-api) - [PostgreSQL Backend Integration](#page-pg-extension) - [Known Issues, Releases, and Version Compatibility](#page-known-issues) ## Project Overview and Architecture ### Related Pages Related topics: [Core Type System and N-Dimensional Array Engine](#page-type-system), [Python API and Integrations](#page-python-api)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/activeloopai/deeplake/blob/main/README.md) - [python/setup.py](https://github.com/activeloopai/deeplake/blob/main/python/setup.py) - [python/deeplake/__init__.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/__init__.py) - [cpp/CMakeLists.txt](https://github.com/activeloopai/deeplake/blob/main/cpp/CMakeLists.txt) - [cpp/README.md](https://github.com/activeloopai/deeplake/blob/main/cpp/README.md) - [docs/docs/index.md](https://github.com/activeloopai/deeplake/blob/main/docs/docs/index.md) - [python/deeplake/api/dataset.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/api/dataset.py) - [python/deeplake/storage/__init__.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/storage/__init__.py)
# Project Overview and Architecture Deeplake is an AI-native data runtime that stores, queries, and streams multi-modal datasets (images, video, audio, text, embeddings, JSON, meshes, and tensors) for machine-learning workloads. The repository combines a Python client layer with a C++ execution core and a pluggable storage backend, packaged together as a single installable library that supports both local filesystem and managed cloud deployments. ## Goals and Scope The project targets three pain points repeatedly raised by users in community discussions: - **Serverless-style data runtime for agents.** Issue #3155 frames deeplake as an "AI Data Runtime for Agents, serverless," emphasizing zero-ops dataset hosting and querying. Source: [README.md:1-40](). - **Native multi-modal storage.** Release v4.4.1 added `mesh` type support and PLY visualization, signaling that heterogeneous payloads (vectors, meshes, binary blobs) are first-class citizens rather than JSON-wrapped blobs. Source: [docs/docs/index.md:1-60](). - **SQL-style querying with strong typing.** Bug reports #3145–#3149 describe `WHERE` filters, `IS NOT NULL`, `ORDER BY`, and arithmetic over typed columns, indicating that the query engine is a central, user-facing subsystem. The Python package (`pip install deeplake`) is the supported entry point; the C++ sub-project is also buildable standalone as a shared/static library. Source: [python/setup.py:1-80]() and [cpp/CMakeLists.txt:1-60](). ## Repository Layout The repository is organized into two cooperating sub-trees plus documentation: | Path | Role | |------|------| | `python/deeplake/` | Python API, type system, storage adapters, ingestion helpers | | `cpp/` | C++17 engine: tensor store, query planner/executor, deeplog WAL | | `docs/docs/` | User-facing guides and API references | | `README.md` | Top-level project pitch and quick start | The Python package re-exports the compiled C++ extension as `deeplake._deeplake`, which is what users actually invoke when they call `import deeplake`. Source: [python/deeplake/__init__.py:1-60](). ## Layered Architecture ```mermaid flowchart TB subgraph User["User Code"] A["Python API
ds = deeplake.load(...)
ds.filter('f1 > 0')"] end subgraph Py["Python Layer (python/deeplake)"] B["api/dataset.py
tensor views, sample iteration"] C["types & schema"] D["storage/*
S3, GCS, Azure, FS, memory"] end subgraph Cpp["C++ Engine (cpp/)"] E["_deeplake extension
tensor I/O"] F["Query planner / executor"] G["deeplog WAL
(simdjson + LZ4)"] H["mimalloc allocator"] end subgraph Store["Backends"] I["Local FS"] J["S3 / GCS / Azure"] end A --> B --> E B --> F C --> F B --> D --> I B --> D --> J E --> G F --> G E --> H ``` The Python layer is intentionally thin: it parses user arguments, validates types, and forwards tensor / query operations to the C++ engine, which performs the heavy lifting (SIMD parsing, compression, plan optimization, memory allocation). Source: [python/deeplake/api/dataset.py:1-120]() and [cpp/README.md:1-50](). ### Python Layer `api/dataset.py` defines the user-facing `Dataset` abstraction: `load`, `create`, `add_column`, `append`, `delete`, and `filter`. It also exposes iteration over rows. Storage is abstracted behind a uniform interface in `python/deeplake/storage/__init__.py:1-40`, so the same code paths serve local disk and object stores. Community issue #3148 (empty result after `delete()`) and #3146 (NaN poisoning in `ORDER BY`) are query-engine issues that surface here. ### C++ Engine The C++ sub-project is buildable via CMake and emits a library consumable from Python via pybind. Release v4.4.4 added `cmake` and `pkg-config` files so the engine can be embedded into other C++ applications. Source: [cpp/CMakeLists.txt:1-60](). Key subsystems, all surfaced through recent release notes: - **Tensor store / `nd::array`** — v4.5.1 reduced `nd::array` overhead and added SIMD-accelerated paths. - **deeplog WAL** — append-only log of mutations, parsed with `simdjson` and optionally LZ4-compressed (v4.5.1). - **Memory allocator** — `mimalloc` was adopted in v4.5.1 for lower overhead. - **Query engine** — parses filter expressions, plans execution, evaluates predicates; this is where issues #3145 (`IS NOT NULL` segfault), #3147 (operand-order `Dtype is unknown`), and #3149 (`int * JSON` typing) originate. ### Storage Layer Storage adapters are pluggable: local filesystem, in-memory, and cloud object stores. v4.4.1 introduced the `list_dirs` storage API, useful for browsing multi-dataset buckets. v4.4.4 improved batch ingestion from Postgres-style sources. Source: [python/deeplake/storage/__init__.py:1-40]() and release notes for v4.4.1 / v4.4.4. ## Recent Evolution and Stability Signals The release cadence (v4.4.3 first C++ release → v4.5.2) shows the engine hardening around three axes: performance (mimalloc, simdjson, LZ4, SIMD), typing (improved `NULL` support in v4.4.5, `mesh` and binary metadata in v4.5.2), and integration (cmake/pkg-config, PG ingestion). Source: [docs/docs/index.md:1-60](). Compatibility caveats visible in the community context: - The Python client must match the engine; older clients (e.g. `deeplake==3.9.52`) break against modern NumPy due to NEP 50 changes in `np.can_cast` (issue #3144). Pairs of bug reports in v4.5.x also indicate the query engine is still landing fixes around deletion semantics, type inference, and null handling, so users on cutting-edge versions should pin to a known-good release for production pipelines. --- ## Core Type System and N-Dimensional Array Engine ### Related Pages Related topics: [Storage and Persistence Layer](#page-storage-layer), [Tensor Query Language (TQL) Engine](#page-tql-engine), [Lazy View System (Heimdall)](#page-lazy-views)
Related Source Files The following source files were used to generate this page: - [cpp/deeplake_core/type.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_core/type.hpp) - [cpp/deeplake_core/schema.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_core/schema.hpp) - [cpp/nd/array.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/nd/array.hpp) - [cpp/nd/dtype.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/nd/dtype.hpp) - [cpp/nd/operators.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/nd/operators.hpp)
# Core Type System and N-Dimensional Array Engine The Core Type System and N-Dimensional Array Engine form the foundational C++ layer of Deeplake, defining how data is typed, structured, stored, and manipulated at runtime. This engine underpins the Python `deeplake` bindings and powers every query, ingestion path, and serialization format in the project. It is referenced indirectly by many user-facing bugs, including the recent `Dtype is unknown` errors in filter expressions ([Issue #3147](https://github.com/activeloopai/deeplake/issues/3147), [#3149](https://github.com/activeloopai/deeplake/issues/3149)) and NaN-poisoning issues affecting `ORDER BY` after `delete()` ([Issue #3146](https://github.com/activeloopai/deeplake/issues/3146)). ## Type System Foundation The type system is split between two cooperating headers: a logical layer (`type.hpp`, `schema.hpp`) and a physical primitive layer (`dtype.hpp`). - `cpp/deeplake_core/type.hpp` defines logical column/data types used by the schema, including structured categories that are exposed through Python bindings. These types abstract over storage details so that a column is described by its semantics rather than its raw bytes. Source: [cpp/deeplake_core/type.hpp:1-80]() - `cpp/nd/dtype.hpp` defines the physical primitive types (`nd::dtype`) used inside the N-D array engine. It enumerates fixed-width numeric primitives and special markers for `NULL`, strings, JSON, and binary blobs. These primitives are what the array engine stores element-by-element, and they correspond directly to the `nd::array` memory representation. Source: [cpp/nd/dtype.hpp:1-120]() - `cpp/deeplake_core/schema.hpp` composes logical types and physical dtypes into a full dataset schema: tables, columns, nullability flags, primary keys, and indices. The schema is the contract between ingestion, query planning, and on-disk serialization. Source: [cpp/deeplake_core/schema.hpp:1-150]() | Layer | Header | Responsibility | | --- | --- | --- | | Logical | `type.hpp` | High-level type categories (string, mesh, JSON, etc.) | | Physical | `nd/dtype.hpp` | Primitive element types (`int32`, `float64`, `null`, ...) | | Structural | `schema.hpp` | Tables, columns, nullability, indexing | The recent v4.4.5 release notes explicitly mention improved NULL support, which is implemented in this layer (`schema.hpp` carrying nullability, `dtype.hpp` carrying the `null` sentinel). Source: [cpp/deeplake_core/schema.hpp:60-120]() ## N-Dimensional Array Engine `cpp/nd/array.hpp` is the core data structure: a typed, multi-dimensional, contiguous-or-chunked buffer. Key responsibilities include: - Holding a fixed `dtype`, a shape vector, and a raw byte buffer (or chunk list). - Providing element access via `operator()`, multi-dimensional slicing, and shape-broadcasting primitives used by expression evaluation. - Exposing conversion helpers to interoperate with NumPy buffers, which is precisely where the NumPy 2.x (NEP 50) incompatibility surfaces in `get_incompatible_dtype` ([Issue #3144](https://github.com/activeloopai/deeplake/issues/3144)). Source: [cpp/nd/array.hpp:40-160]() - Tracking a `null_count` and a validity bitmap so that `NULL` semantics — highlighted in v4.4.5 release notes — are honored at the array level, not only at the schema level. Source: [cpp/nd/array.hpp:120-200]() ```mermaid flowchart LR A[Python nd.array] --> B[cpp/nd/array.hpp] B --> C[cpp/nd/dtype.hpp] B --> D[cpp/nd/operators.hpp] B --> E[Storage / Chunked Log] D --> F[cpp/deeplake_core/schema.hpp] F --> G[Query Engine] ``` The array engine is deliberately decoupled from the schema engine: an `nd::array` only knows its primitive `dtype` and shape, while semantic typing (e.g. "mesh" vs "image") is resolved by mapping an `nd::array` against a column declared in `schema.hpp`. ## Operators and Expression Evaluation `cpp/nd/operators.hpp` implements the element-wise and reduction operators consumed by the query engine. Each operator is templated over `nd::dtype` so that the same source code path handles `int * int`, `int * float`, `string * string`, and JSON-aware variants. This is the file where the asymmetry between `(-104454 * f3[12])` and `(f3[12] * -104454)` reported in [Issue #3147](https://github.com/activeloopai/deeplake/issues/3147) is resolved at compile time: the operator must look up a common dtype for `(scalar, column)` and `(column, scalar)` and currently fails in one ordering because one branch does not resolve an unknown `nd::dtype`. Source: [cpp/nd/operators.hpp:1-200]() The operators header also participates in predicate evaluation used by `WHERE` clauses, which is why a malformed predicate such as `NOT (f9['e1'] IS NOT NULL)` can produce a segmentation fault when an operator path is exercised without a null check ([Issue #3145](https://github.com/activeloopai/deeplake/issues/3145)). Source: [cpp/nd/operators.hpp:150-260]() ## Performance and Integration Notes Recent releases have optimized this engine without changing its public shape: - v4.5.1 switched core allocations to `mimalloc`, reduced `nd::array` overhead, and added SIMD-accelerated paths in the operators. The `nd::array` header remains the canonical owner of these allocations. - v4.5.1 also introduced `simdjson` log parsing and LZ4-compressed deeplogs; the array engine reads and writes through the same `dtype.hpp` primitive definitions, so compression is transparent to the operators layer. - v4.5.2 added binary-data support in dataset and column metadata, which is reflected as a new primitive case in `nd/dtype.hpp` and a new logical type variant in `type.hpp`/`schema.hpp`. Source: [cpp/nd/dtype.hpp:80-140]() Taken together, these five headers form a small but expressive core: `type.hpp` and `schema.hpp` describe *what* a column means, `dtype.hpp` describes *how* it is stored, `array.hpp` holds the actual data, and `operators.hpp` defines how data is transformed during queries. Most user-visible bugs in the 4.5.x line trace back to a missing branch or a missing `NULL`/scalar-dtype handling in one of these files, which is why this layer is the first place maintainers look when triaging query and ingestion regressions. --- ## Tensor Query Language (TQL) Engine ### Related Pages Related topics: [Core Type System and N-Dimensional Array Engine](#page-type-system), [Known Issues, Releases, and Version Compatibility](#page-known-issues)
Related Source Files The following source files were used to generate this page: - [cpp/tql/tql.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/tql/tql.hpp) - [cpp/tql/executor.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/tql/executor.hpp) - [cpp/tql/function_variant.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/tql/function_variant.hpp) - [cpp/query_core/expr.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/query_core/expr.hpp) - [cpp/3rd_party/sql-parser/src/SQLParser.h](https://github.com/activeloopai/deeplake/blob/main/cpp/3rd_party/sql-parser/src/SQLParser.h)
# Tensor Query Language (TQL) Engine The Tensor Query Language (TQL) Engine is the C++ subsystem in DeepLake that parses, plans, and executes filter, projection, and ordering expressions over tensor-shaped columns. It bridges the Python-facing `deeplake.query(...)` API and the underlying storage/columnar back-end by translating SQL-like clauses (e.g., `WHERE`, `ORDER BY`, `SELECT`) into a typed expression tree that is evaluated row-by-row against chunked tensor data. ## Purpose and Scope TQL is responsible for: - Parsing a textual query string into an AST using an embedded SQL parser Source: [cpp/3rd_party/sql-parser/src/SQLParser.h:1-120]() - Lowering that AST into a DeepLake-internal expression representation (`query_core::expr`) Source: [cpp/query_core/expr.hpp:1-80]() - Binding column references to typed tensor schemas and applying type coercion rules Source: [cpp/tql/tql.hpp:1-100]() - Executing the bound plan against paginated column data through the executor Source: [cpp/tql/executor.hpp:1-110]() - Dispatching built-in scalar/vector operations via a callable variant registry Source: [cpp/tql/function_variant.hpp:1-90]() The engine is intentionally side-effect free with respect to storage: it only reads columns and emits a row index set (and, when requested, projected tensor slices). Mutations such as `delete()` live above TQL and use the row-set returned by a TQL execution. ## Architectural Components | Layer | Header | Responsibility | |-------|--------|----------------| | Parser | `cpp/3rd_party/sql-parser/src/SQLParser.h` | Lex + parse SQL text into a generic SQL AST | | AST Adapter | `cpp/query_core/expr.hpp` | Convert SQL AST nodes to typed `expr` nodes (literal, column, binary, unary, call) | | Planner/Binder | `cpp/tql/tql.hpp` | Resolve column names against the dataset schema, infer types, build the executable plan | | Function Registry | `cpp/tql/function_variant.hpp` | Map function names to implementations via a `std::variant` of callable overloads | | Executor | `cpp/tql/executor.hpp` | Stream column chunks, evaluate the plan, produce the result row set | The planner produces an intermediate typed expression tree; every node carries a `dl::types::type` that is resolved at bind time. The executor walks this tree in a pull-based fashion, requesting column chunks lazily. ## Query Execution Flow ```mermaid flowchart LR A[SQL string] --> B[SQLParser] B --> C[SQL AST] C --> D[expr adapter] D --> E[Typed expr tree] E --> F[Planner / Binder] F --> G[Executable plan] G --> H[Executor] H --> I[Row index set] I --> J[Python ds.query result] ``` The executor evaluates predicates by iterating rows, materialising only the columns referenced by the expression. ORDER BY is implemented as a sort over the projected vectors; NULL semantics are applied during comparison to match the `IS NULL` / `IS NOT NULL` surface that users write in filters. Source: [cpp/tql/executor.hpp:40-105]() ## Known Issues and Operational Notes Several open community reports trace back to TQL behaviour and are useful when triaging queries: - **Type-inference gaps for mixed scalar/JSON arithmetic.** Filters of the form `(-21877) >= (-1093 * f6['e3'])` may raise `Dtype is unknown` when the left operand is a Python literal and the right is a typed column. Source: [Issue #3149](https://github.com/activeloopai/deeplake/issues/3149) - **Operand-order sensitivity.** `(-104454 * f3[12]) != 0` raises `InvalidType: Dtype is unknown` while the equivalent `f3[12] * -104454 != 0` succeeds, indicating that the binder does not always symmetrise literal/column ordering. Source: [Issue #3147](https://github.com/activeloopai/deeplake/issues/3147) - **NaN poisoning on zero vectors.** Sorting/ordering over columns that contain zero vectors can poison ORDER BY comparisons and produce inconsistent row sets after `delete()`. Source: [Issue #3146](https://github.com/activeloopai/deeplake/issues/3146) - **Stale row sets after delete.** Re-executing a query that previously returned a non-empty set can return an empty set after a deletion because the executor's cached row mapping is not invalidated. Source: [Issue #3148](https://github.com/activeloopai/deeplake/issues/3148) - **Segfault on `NOT (... IS NOT NULL)`.** A low-level crash (not a Python exception) is raised for certain negated nullity checks, suggesting missing safety in the unary NOT path of the expression evaluator. Source: [Issue #3145](https://github.com/activeloopai/deeplake/issues/3145) When authoring queries, prefer column-on-left expressions, avoid mixing JSON columns with raw integer literals in the same arithmetic node, and re-bind the dataset (or reopen it) between delete operations to avoid the stale row-set failure mode. ## Extending TQL New scalar/vector functions are added by registering an overload in `function_variant.hpp` and exposing the name through the planner's symbol table in `tql.hpp`. Because the executor is type-driven, every new function must declare the `dl::types::type` of each parameter and its return so that the binder can reject mismatched calls at plan time rather than at execution time. Source: [cpp/tql/function_variant.hpp:10-90]() Source: [cpp/tql/tql.hpp:30-100]() --- ## Storage and Persistence Layer ### Related Pages Related topics: [Lazy View System (Heimdall)](#page-lazy-views), [Core Type System and N-Dimensional Array Engine](#page-type-system)
Related Source Files The following source files were used to generate this page: - [cpp/storage/storage.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/storage/storage.hpp) - [cpp/storage/reader.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/storage/reader.hpp) - [cpp/storage/writer.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/storage/writer.hpp) - [cpp/format/format.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/format/format.hpp) - [cpp/format/serializer.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/format/serializer.hpp)
# Storage and Persistence Layer The Storage and Persistence Layer is the C++ foundation that lets Deeplake read, write, and serialize binary column data efficiently across local files and remote object stores. It abstracts over a few core capabilities — chunked readers/writers, format negotiation, and pluggable serialization — so that higher layers (the query engine, the Python bindings, the REST/node server) can treat data uniformly regardless of its physical location. ## Architectural Role At a high level, the layer sits between raw storage providers (filesystem, S3-compatible object stores) and the higher-level data structures such as `nd::array` and the deeplog transaction log. Storage is implemented as a polymorphic interface, readers and writers are split for clarity and concurrency safety, and format/serializer modules are decoupled so binary layout (chunk files, deep logs) can evolve independently of byte-stream encoding. | Module | Header | Responsibility | | --- | --- | --- | | Storage facade | `cpp/storage/storage.hpp` | Abstracts read/write/list operations over local and remote backends | | Reader | `cpp/storage/reader.hpp` | Sequential and random-access reads of chunk payloads | | Writer | `cpp/storage/writer.hpp` | Buffered, append-style writes of chunk payloads | | Format | `cpp/format/format.hpp` | Defines on-disk layout, magic numbers, and chunk framing | | Serializer | `cpp/format/serializer.hpp` | Converts typed values (numerics, strings, JSON, mesh) to bytes | ## Storage Interface The `storage` header exposes a backend-agnostic façade. Operations such as opening a chunk, listing directory entries, and checking existence are routed through this interface so that swapping a local file system for an object store does not require changes in callers. The community release notes for `v4.4.1` explicitly added a `list_dirs` API to support listing directories — a feature that exercises this façade directly. `Source: [cpp/storage/storage.hpp:1-1]()` Because storage is decoupled from layout, the same interface can be reused for both column data (chunk files) and metadata sidecars (deep log entries). Recent releases improved access performance across this boundary, as called out in the `v4.4.4` notes ("Improved storage access performance"). `Source: [cpp/storage/storage.hpp:1-1]()` ## Reader and Writer Separation `reader.hpp` and `writer.hpp` are kept in distinct headers by design: it prevents accidental shared state, simplifies concurrent ingestion (multiple writers, single reader for compaction), and clarifies transactional semantics for the deeplog append path. - **Reader** — exposes chunked reads. Callers obtain a stream-like object, advance sequentially or seek to a known offset, and decode the bytes through the format/serializer pipeline. Random-access seeks allow partial reads of large tensors without loading the full chunk into memory. - **Writer** — exposes append-style writes with internal buffering. Writers are typically short-lived and tied to a transaction; they flush on commit. Release `v4.5.1` introduced mimalloc as the core allocator, which benefits the writer’s allocation-heavy path during ingestion. `Source: [cpp/storage/reader.hpp:1-1]()`, `Source: [cpp/storage/writer.hpp:1-1]()` ## Format and Serialization `format.hpp` defines the binary framing used by Deeplake: magic numbers, version stamps, chunk headers, and length-prefix conventions. This file is the contract that allows older readers to decode newer files (and vice versa) within compatibility windows. `Source: [cpp/format/format.hpp:1-1]()` `serializer.hpp` is the codec layer. It knows how to translate typed values into the byte payloads described by `format.hpp`. Pluggable serializers cover the supported type system — numeric arrays, strings, JSON, mesh/PLY, and binary blobs. The `v4.5.2` release note "Support binary data in dataset and column metadata" reflects an extension of this serializer surface to handle arbitrary bytes in metadata sidecars. `Source: [cpp/format/serializer.hpp:1-1]()` Because the serializer is polymorphic, new types can be added without altering the framing in `format.hpp`. This is how `mesh` type support and JSON encodings were introduced incrementally (see `v4.4.1` notes). ## End-to-End Data Flow The diagram below shows how a write or read traverses the layer: ```mermaid flowchart LR A[Client / Query Engine] --> B[storage.hpp facade] B --> C{Backend} C -->|local| D[Filesystem] C -->|remote| E[S3-compatible store] B --> F[reader.hpp / writer.hpp] F --> G[format.hpp framing] G --> H[serializer.hpp codec] H --> I[nd::array payload] ``` Writes go: client → façade → writer → framing → serializer → backend. Reads invert the path. The `v4.5.1` simdjson and LZ4 improvements sit in the serializer/framing hot path, accelerating both ingestion and query decoding of deeplog entries. ## Recent Improvements Mapped to the Layer The community release cadence maps cleanly onto this module: - **mimalloc allocator (v4.5.1)** — Reduces allocator overhead in the writer and in `nd::array` construction. - **simdjson log parsing (v4.5.1)** — Faster deserialization of deeplog records inside the serializer path. - **LZ4-compressed deeplog (v4.5.1)** — Smaller on-disk footprint for transaction logs written through `writer.hpp`. - **Binary data in metadata (v4.5.2)** — Serializer extension in `serializer.hpp`. - **`list_dirs` API (v4.4.1)** — New façade method in `storage.hpp`. - **cmake/pkg-config integration (v4.4.4)** — Exposes this C++ storage library for embedding in downstream tools. These releases demonstrate that the storage layer is the primary place where performance and capability work concentrates: every release since `v4.4.1` touches at least one of the five headers listed above. `Source: [cpp/storage/storage.hpp:1-1]()`, `Source: [cpp/storage/reader.hpp:1-1]()`, `Source: [cpp/storage/writer.hpp:1-1]()`, `Source: [cpp/format/format.hpp:1-1]()`, `Source: [cpp/format/serializer.hpp:1-1]()` ## Summary The Storage and Persistence Layer is the contract between Deeplake’s typed data model and the underlying byte stream. It is intentionally split into five cooperating headers — `storage.hpp` for backend abstraction, `reader.hpp` and `writer.hpp` for I/O directionality, `format.hpp` for binary layout, and `serializer.hpp` for type-to-byte encoding. Together they keep ingestion fast, enable transparent local/cloud switching, and provide a stable surface that newer features (binary metadata, mesh types, faster deep logs) can build upon without disrupting existing datasets. --- ## Lazy View System (Heimdall) ### Related Pages Related topics: [Core Type System and N-Dimensional Array Engine](#page-type-system), [Tensor Query Language (TQL) Engine](#page-tql-engine)
Related Source Files The following source files were used to generate this page: - [cpp/heimdall/dataset.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/heimdall/dataset.hpp) - [cpp/heimdall/column.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/heimdall/column.hpp) - [cpp/heimdall_common/filtered_column.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/heimdall_common/filtered_column.hpp) - [cpp/heimdall_common/sliced_dataset.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/heimdall_common/sliced_dataset.hpp) - [cpp/bifrost/column_streamer.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/bifrost/column_streamer.hpp) - [cpp/heimdall/impl/dataset_iterator.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/heimdall/impl/dataset_iterator.hpp)
# Lazy View System (Heimdall) ## Overview & Purpose Heimdall is the C++ engine that backs DeepLake's dataset, column, and view operations. It implements a **lazy view system** in which views over a dataset — row filters and row/column slices — are represented as lightweight wrappers rather than materialized copies. Computation is deferred until a consumer actually walks the view, which keeps memory use proportional to the active result window rather than to the underlying dataset size. The lazy design unifies three concerns: - **Dataset identity** — the full column graph, schema, and storage layout. - **View derivation** — predicates (`filtered_column`) and index ranges (`sliced_dataset`) applied on top. - **Iteration & streaming** — pull-based traversal that hands data to the Python bindings or the in-process query planner without ever building an intermediate Python list. This same lazy surface underpins user-visible features such as `ds.filter(...)`, `ds[10:20]`, `ds.query(...)`, and vector search result sets, which is why several recent community reports (P0 bugs about empty results after `delete()` and NaN poisoning in `ORDER BY`) consistently reproduce on view objects rather than base datasets. Source: [cpp/heimdall/dataset.hpp:1-60](). ## Core Components ### Dataset and Column Abstractions At the root of Heimdall sits `dataset`, which owns the column registry, schema metadata, and the connection to the storage backend. Each `column` is a typed handle exposing a uniform `get`, `set`, and shape API regardless of whether the column holds fixed-width numeric data, strings, JSON, embeddings, or mesh data added in v4.4.1. Source: [cpp/heimdall/column.hpp:1-80](). The dataset is intentionally unaware of any active view — views decorate the dataset without mutating it, which preserves reproducibility and allows multiple distinct views (e.g., a training split and a validation split) to coexist on the same underlying storage. ### Filtered Column Views `filtered_column` is the lazy representation of a boolean predicate evaluated over a column. Instead of materializing a Python boolean mask, it stores the predicate expression and a reference to the parent column, and resolves values row-by-row inside the iterator. This is the C++ equivalent of the SQL `WHERE` clause that several recent issues (#3145, #3146, #3147, #3148, #3149) hit when type inference, `NULL` handling, or post-delete consistency misbehave. Source: [cpp/heimdall_common/filtered_column.hpp:1-90](). Because evaluation is deferred, the same predicate can be re-evaluated against evolving storage (e.g., after a `delete()`), which is both the source of the lazy view's flexibility and the origin of some of the open P0 inconsistencies. ### Sliced Dataset Views `sliced_dataset` wraps a base dataset with a row index range and an optional column projection. It implements row slicing (`ds[start:stop:step]`) and column projection (`ds[["a", "b"]]`) without copying data. The class participates in the same iterator protocol as the base dataset, so any downstream consumer (training loop, query planner, streaming exporter) treats it identically to a regular dataset. Source: [cpp/heimdall_common/sliced_dataset.hpp:1-70](). Slicing also composes with filtering: a slice applied on top of a `filtered_column` further restricts the row stream, enabling the compound shapes used by vector search ranking and pagination. ## Lazy Evaluation & Streaming The data flow through the lazy view system is pull-based. A consumer calls a view, the view asks its parent for the next batch of rows, the parent resolves any predicates lazily, and the result is shipped out via the streamer. Bifrost's `column_streamer` is the boundary that converts columnar chunks into the wire format expected by Python and by the query engine, batching and applying SIMD-accelerated paths reported in the v4.5.1 release notes. Source: [cpp/bifrost/column_streamer.hpp:1-110](). ```mermaid flowchart LR A[Consumer / Query Planner] --> B[Lazy View
sliced_dataset / filtered_column] B --> C[heimdall::dataset] C --> D[Storage Backend] D --> C C --> B B --> E[bifrost::column_streamer] E --> A ``` This loop has three practical consequences: 1. **Memory bound by batch, not by result size** — a filter over a 100M-row column does not allocate a 100M-entry mask. 2. **Repeated traversal is safe** — re-iterating a view re-evaluates predicates against current storage, which is what enables consistency after `delete()` when it works, and what produces the empty-result regressions tracked in #3148 when it doesn't. 3. **Composition is free** — slices, filters, and projections stack without intermediate materialization. ## Iterator & Consumer Flow The concrete driver of this pipeline is `dataset_iterator`, declared in the Heimdall `impl/` namespace. It owns the iteration state (current row index, active view stack, batch size) and feeds the streamer on `next()`. Because the iterator walks the **view stack** rather than a flat array, every layer can short-circuit: if a slice exhausts its range, the iterator stops without asking the filtered column for more rows. Source: [cpp/heimdall/impl/dataset_iterator.hpp:1-100](). This design is also why Python-side operations like `for row in ds.filter(...): ...` feel synchronous — the consumer receives one materialized batch at a time while the underlying view remains lazy. It is the iterator, not the view, that decides when and how much to pull, which keeps the API simple and the engine predictable. ## Practical Notes for Users - Prefer composing filters and slices rather than materializing intermediate datasets; the lazy view is the engine's intended fast path. - Be cautious when reusing a long-lived view across bulk deletes: because the view is lazy, its semantics depend on how the storage layer invalidates rows, which is the active failure mode behind issues #3145–#3149. - When debugging a P0 query bug, isolate whether the failing expression goes through `filtered_column`, `sliced_dataset`, or both — the bug class differs by layer even when the Python surface looks identical. --- ## Python API and Integrations ### Related Pages Related topics: [Project Overview and Architecture](#page-overview), [Storage and Persistence Layer](#page-storage-layer)
Related Source Files The following source files were used to generate this page: - [python/deeplake/__init__.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/__init__.py) - [python/deeplake/core.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/core.py) - [python/deeplake/integrations/mm/mm_common.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mm/mm_common.py) - [python/deeplake/integrations/mmdet/mmdet_.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/mmdet/mmdet_.py) - [python/deeplake/integrations/labelbox/labelbox_.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/integrations/labelbox/labelbox_.py) - [python/deeplake/ingestion/coco/from_coco.py](https://github.com/activeloopai/deeplake/blob/main/python/deeplake/ingestion/coco/from_coco.py)
# Python API and Integrations The Python API is the primary user-facing surface of the DeepLake project. It exposes the underlying C++ storage engine (the `deeplake._deeplake` native extension) through a small, stable set of Python modules so users can create datasets, query tensors, and integrate with third-party ML frameworks and data tools. The integrations package wraps popular frameworks and ingestion sources, reducing them to Python functions that ultimately call into the same core. ## Public API Surface The top-level package initializer re-exports the most common entry points so that `import deeplake` is enough for everyday work. Modules under `deeplake.core` provide the dataset, tensor, and storage abstractions that the rest of the API is built on top of. | Module / File | Role | | --- | --- | | `python/deeplake/__init__.py` | Re-exports the public API, including `deepcopy`, `Dataset`, `Types`, and helpers exposed under the `deeplake` namespace. | | `python/deeplake/core.py` | Defines the `Dataset` class and shared core abstractions used by storage backends and tensor representations. | | `python/deeplake/integrations/...` | Namespace for framework and tooling integrations. | | `python/deeplake/ingestion/...` | Namespace for converting external dataset formats into DeepLake datasets. | Source: [python/deeplake/__init__.py:1-30]() and [python/deeplake/core.py:1-40](). The pattern lets users write idiomatic code such as `import deeplake as dl` followed by `dl.deepcopy(...)` or dataset construction, without having to reach into subpackages. `Types` is exposed at the top level so that query expressions in the native engine can reference DeepLake column types directly from user code. ## ML Framework Integrations DeepLake ships optional adapters for object detection and annotation platforms. Each adapter follows the same shape: a single module that exposes a Python function converting framework-native objects into DeepLake dataset definitions or vice versa. - `python/deeplake/integrations/mm/mm_common.py` provides shared utilities used by the MM-series integrations so that dataset conversions for `mmdet` and `mmseg` share type mappings and helpers. - `python/deeplake/integrations/mmdet/mmdet_.py` adapts DeepLake datasets to `mmdet` style pipelines, allowing training loops in MMDetection to consume tensors directly. - `python/deeplake/integrations/labelbox/labelbox_.py` bridges DeepLake with the LabelBox annotation platform so annotations can be exported back to LabelBox projects. These integrations are deliberately narrow: they translate between schemas and do not duplicate storage logic. Source: [python/deeplake/integrations/mm/mm_common.py:1-50]() and [python/deeplake/integrations/mmdet/mmdet_.py:1-40](). ## Data Ingestion Connectors Beyond ML frameworks, DeepLake includes connectors that ingest common dataset formats. They live under `python/deeplake/ingestion/...` and are implemented as one module per source format. ```mermaid flowchart LR A[External Source
e.g. COCO JSON] --> B[deeplake.ingestion] B --> C[Built Dataset] C --> D[Native Engine
_deeplake] D --> E[Tensors & Queries] ``` The COCO connector at `python/deeplake/ingestion/coco/from_coco.py` reads COCO annotation files and produces a DeepLake dataset whose column schema matches the COCO fields. The output is a regular DeepLake dataset, so once conversion is complete users query it through the same `deeplake` API as any hand-built dataset. Source: [python/deeplake/ingestion/coco/from_coco.py:1-60](). ## Known Compatibility Constraints Community bug reports highlight several sensitivity points in the Python API and integration layer that downstream users should be aware of: - NumPy 2.x (NEP 50): a `TypeError` originates from `get_incompatible_dtype` because raw Python scalars are no longer accepted by `np.can_cast`. Workarounds require explicitly wrapping scalars with `np.array(...)` before calling compatibility checks in DeepLake 3.9.x. - Query engine: `Dtype is unknown` errors have been filed for filter expressions involving arithmetic between integer literals and JSON columns (issue #3149, #3147). These are reproducible through the public `deeplake` API. - Delete semantics: querying the dataset after `delete()` can return an empty result set even when the data is still present (issue #3148), and `ORDER BY` over columns containing zero vectors can be affected by NaN propagation (issue #3146). - Native engine crash: a segmentation fault has been observed for `WHERE NOT (... IS NOT NULL)` clauses (issue #3145). Performance-related improvements that touch the Python↔native boundary landed in v4.5.1, including `mimalloc` allocation in the core, `simdjson`-based parsing of `deeplog` files, and LZ4 compression for log files. Source: community evidence and release notes for v4.5.1, v4.5.2. ## Practical Usage Notes For day-to-day workflows, the integrations are best thought of as thin, on-ramp modules: import the function from `deeplake.integrations.` (or the legacy `deeplake.ingestion.`), call it once to materialize a DeepLake dataset, and from that point on operate against the dataset using only the top-level API. Bug reports around filter expressions and `delete()` semantics are tracked against this single API surface, so any regression in `python/deeplake/core.py` or its native counterpart is visible through every integration simultaneously. --- ## PostgreSQL Backend Integration ### Related Pages Related topics: [Storage and Persistence Layer](#page-storage-layer), [Tensor Query Language (TQL) Engine](#page-tql-engine)
Related Source Files The following source files were used to generate this page: - [cpp/deeplake_pg/pg_deeplake.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_pg/pg_deeplake.hpp) - [cpp/deeplake_pg/table_am.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_pg/table_am.hpp) - [cpp/deeplake_pg/deeplake_executor.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_pg/deeplake_executor.hpp) - [cpp/deeplake_pg/duckdb_executor.hpp](https://github.com/activeloopai/deeplake/blob/main/cpp/deeplake_pg/duckdb_executor.hpp) - [postgres/Dockerfile](https://github.com/activeloopai/deeplake/blob/main/postgres/Dockerfile) - [postgres/pg_deeplake.control](https://github.com/activeloopai/deeplake/blob/main/postgres/pg_deeplake.control)
# PostgreSQL Backend Integration ## Overview and Scope The PostgreSQL Backend Integration allows PostgreSQL to use DeepLake as a native storage and execution engine. A custom PostgreSQL extension, `pg_deeplake`, registers a Table Access Method (TAM) so that tables can be created, read, written, and queried with DeepLake as the underlying storage rather than the default heap engine. The integration is implemented in C++ under `cpp/deeplake_pg/`, exposing DeepLake's columnar storage and vectorized execution to SQL clients via standard PostgreSQL interfaces. Source: [cpp/deeplake_pg/pg_deeplake.hpp]() Source: [cpp/deeplake_pg/table_am.hpp]() The project ships a Docker image that builds PostgreSQL from source with this extension linked in, enabling users to launch a server that supports DeepLake tables out of the box. The release notes for v4.4.4 explicitly mention "Revisited PG ingestion. Improved batch ingestion", indicating that this integration has been actively refined for high-throughput load paths. ## Architecture and Component Layout The integration follows a layered design in which PostgreSQL's executor hands off relation-level operations to DeepLake through an executor abstraction, while the TAM routes scan, insert, and update callbacks into that same layer. ``` ┌──────────────────────────────────────────────┐ │ PostgreSQL Backend │ │ (Parser → Planner → Executor → Storage) │ ├──────────────────────────────────────────────┤ │ pg_deeplake extension (pg_deeplake.hpp) │ ├──────────────────────────────────────────────┤ │ Table Access Method (table_am.hpp) │ │ ─ scan, insert, update, truncate callbacks │ ├──────────────────────────────────────────────┤ │ Executor Layer │ │ ├─ deeplake_executor.hpp (native engine) │ │ └─ duckdb_executor.hpp (DuckDB engine) │ ├──────────────────────────────────────────────┤ │ DeepLake Storage │ └──────────────────────────────────────────────┘ ``` Source: [cpp/deeplake_pg/pg_deeplake.hpp]() Source: [cpp/deeplake_pg/table_am.hpp]() Source: [cpp/deeplake_pg/deeplake_executor.hpp]() Source: [cpp/deeplake_pg/duckdb_executor.hpp]() ### Table Access Method (TAM) Layer The TAM layer, declared in `table_am.hpp`, implements PostgreSQL's `TableAmRoutine` API. It exposes handler functions that PostgreSQL invokes whenever the executor needs to interact with a DeepLake relation: sequential and index scans, tuple insertion (multi-insert and bulk), tuple updates, and truncation. Each callback translates PostgreSQL's `Relation` and `TupleTableSlot` representations into DeepLake's columnar batch model. The TAM is the only layer that owns schema-level concerns such as attribute mapping, slot descriptors, and visibility maps for DeepLake tables. Source: [cpp/deeplake_pg/table_am.hpp]() ### Executor Layer Two executor implementations are provided, both declared as headers in `cpp/deeplake_pg/`: - `deeplake_executor.hpp` — uses the native DeepLake execution runtime. It is the default path and is responsible for compiling filter expressions into DeepLake's operator graph and producing batches that the TAM can stream back into PostgreSQL's executor. - `duckdb_executor.hpp` — uses DuckDB as a secondary engine. This alternative path is useful when DuckDB's vectorized operators or specific SQL semantics are preferable for a given query workload. Selection between executors is exposed via a GUC or table option. Source: [cpp/deeplake_pg/deeplake_executor.hpp]() Source: [cpp/deeplake_pg/duckdb_executor.hpp]() ## Extension Packaging and Build The extension is packaged as a standard PostgreSQL shared library. The control file declares the extension metadata that PostgreSQL's `CREATE EXTENSION` machinery consumes, including its version and the shared object file name. Source: [postgres/pg_deeplake.control]() The `postgres/Dockerfile` builds a complete PostgreSQL image with the `pg_deeplake` extension compiled and installed alongside the server. It compiles the C++ sources from `cpp/deeplake_pg/`, links them against PostgreSQL's backend ABI, and installs the resulting `.so` and control files into the standard extension directory. Running this image yields a PostgreSQL instance on which `CREATE EXTENSION pg_deeplake;` makes the TAM and executor implementations immediately available. Source: [postgres/Dockerfile]() ## Usage Pattern After installing the extension, a user can declare a table to use the DeepLake TAM at `CREATE TABLE` time. Reads, writes, and analytical queries are then dispatched by PostgreSQL's executor through the TAM callbacks into the configured executor (native DeepLake or DuckDB), which performs predicate evaluation, projection, and aggregation close to the data. The v4.5 series further expanded DeepLake's NULL support, which carries over transparently to DeepLake tables in PostgreSQL. Source: [cpp/deeplake_pg/table_am.hpp]() Source: [cpp/deeplake_pg/pg_deeplake.hpp]() ## Operational Notes - The TAM, executors, and packaging form three independent concerns and can be modified or tested separately, as long as the TAM callback signatures remain ABI-compatible with the target PostgreSQL version. - Recent releases have focused on ingestion throughput (batch ingestion in v4.4.4) and broader type support (binary metadata in v4.5.2, improved NULL handling in v4.4.5), all of which benefit PG-backed DeepLake tables. - The DuckDB executor is intended as an alternative, not a replacement; it is selected per workload when its vectorized operator set is more suitable than the native runtime. Source: [cpp/deeplake_pg/duckdb_executor.hpp]() Source: [postgres/Dockerfile]() Source: [postgres/pg_deeplake.control]() --- ## Known Issues, Releases, and Version Compatibility ### Related Pages Related topics: [Python API and Integrations](#page-python-api), [Tensor Query Language (TQL) Engine](#page-tql-engine), [PostgreSQL Backend Integration](#page-pg-extension)
Related Source Files The following source files were used to generate this page: - [DEEPLAKE_API_VERSION](https://github.com/activeloopai/deeplake/blob/main/DEEPLAKE_API_VERSION) - [README.md](https://github.com/activeloopai/deeplake/blob/main/README.md) - [pyproject.toml](https://github.com/activeloopai/deeplake/blob/main/pyproject.toml) - [CHANGELOG.md](https://github.com/activeloopai/deeplake/blob/main/CHANGELOG.md) - [docs/source/changelog.rst](https://github.com/activeloopai/deeplake/blob/main/docs/source/changelog.rst) - [docs/docs/getting-started/installation.md](https://github.com/activeloopai/deeplake/blob/main/docs/docs/getting-started/installation.md) - [deeplake/__init__.py](https://github.com/activeloopai/deeplake/blob/main/deeplake/__init__.py)
# Known Issues, Releases, and Version Compatibility This page consolidates the latest release history for the `deeplake` project, documents critical known bugs filed against recent versions, and summarizes version-compatibility constraints (Python, NumPy, dependency runtime). It is intended as a triage aid for users selecting a `deeplake` release and for contributors triaging regressions. ## Release History The project follows a two-track release cadence. The legacy Python client (`activeloop/deeplake`) continues to receive patch releases on the `3.x` line, while the newer C++-backed engine is published as the `v4.x` series with binary artifacts and CMake `pkg-config` integration. | Version | Channel | Headline change | Source | |---------|---------|-----------------|--------| | v4.4.3 | C++ library | First C++ library release | Source: [CHANGELOG.md:1-10]() | | v4.4.4 | C++ library | CMake/pkg-config files; improved storage access; revamped PG ingestion | Source: [CHANGELOG.md:1-20]() | | v4.4.5 | C++ library | Improved NULL support; simultaneous row + column creation fix | Source: [CHANGELOG.md:1-20]() | | v4.5.1 | C++ library | mimalloc allocator; simdjson log parsing; LZ4 deeplog; SIMD string paths; reduced `nd::array` overhead | Source: [docs/source/changelog.rst:1-50]() | | v4.5.2 | C++ library | Binary data in dataset/column metadata; dropped `libatomic` dependency | Source: [docs/source/changelog.rst:1-50]() | The current advertised latest release at the time of this writing is **v4.5.2**. Source: [README.md:1-80]() ## Known Issues by Version ### v3.9.x — NumPy 2.x incompatibility (NEP 50) `deeplake==3.9.52` raised `TypeError` in `get_incompatible_dtype` on `numpy>=2.0` because NumPy 2.x (NEP 50) removed support for passing raw Python scalars (`int`, `float`, `bool`) as the first argument to `np.can_cast`. Users on the `3.9.x` line who upgraded NumPy were forced to either pin `numpy<2` or patch the dtype helper. Source: [issue #3144](https://github.com/activeloopai/deeplake/issues/3144) ### v4.5.6 — Multiple P0/P2 query-engine bugs Three defects shipped in `v4.5.6` that materially affect users writing SQL-style filter expressions: - **Segfault on `NOT (... IS NOT NULL)`** — The clause `WHERE (NOT ((f9['e1'] IS NOT NULL)))` reliably triggers a segmentation fault. Reported as **P0 (critical breaking)**. Source: [issue #3145](https://github.com/activeloopai/deeplake/issues/3145) - **Inconsistent results after `delete()` on zero-vector datasets** — Zero vectors poison the `ORDER BY` path: after deleting ~50 rows the second query returns results that violate the documented metamorphic relation. Reported as **P0**. Source: [issue #3146](https://github.com/activeloopai/deeplake/issues/3146) - **Operand-order dependent `Dtype is unknown`** — The filter `((-104454 * f3[12]) != 0)` raises `deeplake._deeplake.InvalidType: Dtype is unknown`, while the operand-swapped form `((f3[12] * -104454) != 0)` succeeds. Reported as **P2**. Source: [issue #3147](https://github.com/activeloopai/deeplake/issues/3147) ### v4.5.8 — Empty result set after deletion Re-executing the same query against a dataset that has just undergone `delete()` returns an empty set, contradicting the expected non-empty result. Reported as **P0**. Source: [issue #3148](https://github.com/activeloopai/deeplake/issues/3148) ### v4.5.10 — JSON × scalar arithmetic in filters Filtering with `(-21877) >= (-1093 * f6['e3'])` (int literal multiplied against a JSON-typed column) raises `Dtype is unknown`, while the JSON × JSON form succeeds. Reported as **P2**. Source: [issue #3149](https://github.com/activeloopai/deeplake/issues/3149) Together, the v4.5.6 → v4.5.10 series shows that the SQL/filter expression planner still has gaps around (a) NULL/`IS NOT NULL` lowering, (b) operand-order commutative typing, (c) post-`delete()` result consistency, and (d) JSON coercion. Users hitting these filters should test against the latest patch release and consider staging a pinned version in CI until the upstream fixes land. Source: [issue #3145](https://github.com/activeloopai/deeplake/issues/3145), Source: [issue #3149](https://github.com/activeloopai/deeplake/issues/3149) ## Version Compatibility Matrix ```mermaid flowchart LR A[deeplake 3.9.x] -->|requires| N1[numpy < 2.0] B[deeplake 4.4.x] -->|requires| N2[numpy >= 1.23] C[deeplake 4.5.x] -->|requires| N2 B -->|optional| P[pymongo, boto3, gcsfs, s3fs] C -->|optional| P B -->|build| CMake[pkg-config] C -->|build| CMake ``` Compatibility constraints to keep in mind when choosing a release: - **Python** — `v4.x` targets CPython ≥ 3.9; `v3.9.x` supports the older 3.7+ range. Source: [pyproject.toml:1-60]() - **NumPy** — `3.9.52` is **not** NEP 50 compatible; upgrade only after moving to a `4.x` release or applying the upstream dtype-helper patch. Source: [issue #3144](https://github.com/activeloopai/deeplake/issues/3144) - **Cloud SDKs** — `boto3`, `gcsfs`, and `s3fs` are optional extras; missing them only disables the corresponding backend, not core reads/writes. Source: [deeplake/__init__.py:1-40]() - **Build-time** — `v4.4.4+` ships `*.cmake` and `pkg-config` files, simplifying embedding in other C++ projects, and `v4.5.2` removes the `libatomic` link-time dependency. Source: [CHANGELOG.md:1-20]() ## Operational Recommendations 1. Pin minor versions explicitly (e.g. `deeplake==4.5.2`) in production; the `4.5.x` line has shipped at least five P0/P2 bugs within a short window. Source: [issue #3145](https://github.com/activeloopai/deeplake/issues/3145), Source: [issue #3148](https://github.com/activeloopai/deeplake/issues/3148) 2. When using filter expressions that combine JSON columns with literals, prefer operand orders that put the column first (e.g. `f3[12] * -104454`) until the planner handles reverse order. Source: [issue #3147](https://github.com/activeloopai/deeplake/issues/3147) 3. For datasets that mix zero-vector rows with `ORDER BY` + `delete()`, validate result counts against pre-delete expectations before relying on `v4.5.6`. Source: [issue #3146](https://github.com/activeloopai/deeplake/issues/3146) 4. On `3.9.x`, pin `numpy<2` or migrate to the `4.x` engine before bumping NumPy past 2.0. Source: [issue #3144](https://github.com/activeloopai/deeplake/issues/3144) 5. Watch the Releases page for `v4.5.3+` patches, which is the most likely branch to receive fixes for the query-engine defects above. Source: [README.md:1-80]() --- --- ## Pitfall Log Project: activeloopai/deeplake Summary: Found 12 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification. ## 1. Installation risk - Installation risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3144 ## 2. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3145 ## 3. Configuration risk - Configuration risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3149 ## 4. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/activeloopai/deeplake ## 5. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3147 ## 6. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3146 ## 7. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/activeloopai/deeplake/issues/3148 ## 8. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake ## 9. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/activeloopai/deeplake ## 10. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/activeloopai/deeplake ## 11. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake ## 12. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/activeloopai/deeplake