# https://github.com/lanterndata/lantern Project Manual Generated at: 2026-07-08 15:44:12 UTC ## Table of Contents - [Lantern Overview and Component Architecture](#page-1) - [lantern_hnsw C Extension Internals](#page-2) - [lantern_cli and lantern_extras Tooling](#page-3) - [Build, Installation, Release, and Known Failure Modes](#page-4) ## Lantern Overview and Component Architecture ### Related Pages Related topics: [lantern_hnsw C Extension Internals](#page-2), [lantern_cli and lantern_extras Tooling](#page-3), [Build, Installation, Release, and Known Failure Modes](#page-4)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/lanterndata/lantern/blob/main/README.md) - [Cargo.toml](https://github.com/lanterndata/lantern/blob/main/Cargo.toml) - [lantern_hnsw/CMakeLists.txt](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/CMakeLists.txt) - [lantern_hnsw/sql/lantern.sql](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/sql/lantern.sql) - [lantern_hnsw/src/hnsw.h](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw.h) - [lantern_hnsw/src/hnsw/usearch_storage.hpp](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/usearch_storage.hpp) - [lantern_hnsw/src/hnsw/delete.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/delete.c) - [lantern_cli/src/main.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/main.rs)
# Lantern Overview and Component Architecture Lantern is an open-source PostgreSQL extension that brings production-grade vector similarity search into the database engine. It combines an in-process C/C++ HNSW implementation with an external Rust daemon that can offload heavy index construction, autotuning, and embedding generation out of the database backend. The project is designed so that users get the ergonomics of standard SQL plus the performance and flexibility of an HNSW (Hierarchical Navigable Small World) ANN index. ## Purpose and Scope Lantern's primary goal is to enable efficient approximate nearest neighbor (ANN) search on vector columns stored inside PostgreSQL. The project exposes: - A native HNSW index access method usable with standard `CREATE INDEX ... USING hnsw` syntax. - Operator classes supporting common distance functions (`l2sq_dist`, `cos_dist`, `hamming_dist`, etc.). - An external indexing path that streams index construction over a TCP socket so memory-heavy builds do not exhaust the PostgreSQL backend. - Optional jobs (autotune, embedding generation, external index builds) that are dispatched to a companion daemon. The repository is intentionally modular: SQL glue and C entry points live under `lantern_hnsw/`, while the long-running background workers and external index server live in the Rust workspace under `lantern_cli/`. Source: [README.md:1-40](). ## Core Components ### PostgreSQL Extension (`lantern_hnsw/`) This directory is the heart of the database integration. It is built with CMake and registered as a PostgreSQL extension. - `lantern_hnsw/CMakeLists.txt` defines the build target, source set, and PostgreSQL include/link flags. - `lantern_hnsw/sql/lantern.sql` registers the data types, operators, operator classes, and the `lantern_hnsw` access method in the catalog. - `lantern_hnsw/src/hnsw.h` declares the HNSW entry points (`hnsw_build`, `hnsw_insert`, `hnsw_search`, `hnsw_handle_delete`) that the index access method calls. - `lantern_hnsw/src/hnsw/delete.c` implements HNSW-aware deletion handling and integrates with PostgreSQL's vacuum infrastructure. Source: [lantern_hnsw/src/hnsw/delete.c:42](). The extension uses the `usearch` library as its underlying SIMD-optimized distance/search engine, wrapped through `lantern_hnsw/src/hnsw/usearch_storage.hpp`. Source: [lantern_hnsw/src/hnsw/usearch_storage.hpp:1-80](). ### Rust Workspace and External Index Daemon (`lantern_cli/`) The Rust workspace provides: - An external index server that receives index build requests over TCP (with optional SSL) and streams back the completed index file. Source: [lantern_cli/src/main.rs:1-120](). - A job runner for autotune and embedding tasks that are feature-gated so deployments can disable them. - A companion CLI that the SQL extension invokes when the `external_indexing` GUC is enabled. Feature gating, introduced in v0.5.0, lets operators turn off embedding generation, autotune, and external indexing independently. Source: [README.md:1-60](). ### SQL Surface and Operator Classes The SQL declarations define what users actually type: - Vector type and `l2sq_dist`, `cos_dist`, `hamming_dist` operators for `<->` distance queries. - The `lantern_hnsw` access method with configurable parameters (`dim`, `ef_construction`, `ef_search`, `M`). - Helper functions to inspect index metadata and to coordinate with the external daemon. Source: [lantern_hnsw/sql/lantern.sql:1-120](). ## High-Level Architecture The diagram below shows how a query or build request flows through the system when external indexing is enabled. ```mermaid flowchart LR A[SQL Client] --> B[PostgreSQL Backend] B --> C[lantern_hnsw Access Method] C -- in-process build --> D[HNSW + usearch] C -- external build --> E[lantern_cli TCP Server] E --> D D --> F[Index File on Disk] C --> G[Catalog & WAL] B --> H[Background Worker / Job] H -- autotune/embed --> E ``` Two important architectural properties follow from this layout: 1. **Single WAL pass for builds**: Recent versions optimized index construction so that node writes go through PostgreSQL's WAL in a single pass rather than buffering the whole graph in memory. 2. **Streaming index selection**: Result sets can be streamed as the HNSW graph is walked, reducing latency for large `LIMIT` queries. ## Known Limitations and Build Considerations From community-reported issues, several practical boundaries are worth recording: - The current extension caps vector dimensions at **2000**, which blocks use of larger embedding models such as OpenAI's `text-embedding-3-large` (3072 dims). Source: [lantern_hnsw/sql/lantern.sql:1-60](). - Builds against PostgreSQL 18 beta currently fail because `vacuum_delay_point()` requires a new `is_analyze` argument that older Lantern sources do not pass. Source: [lantern_hnsw/src/hnsw/delete.c:42](). - macOS users have reported intermittent failures (`os error 22`) when the external index path is enabled on Homebrew-built PostgreSQL 16; the recommended workaround is to rebuild without `external_indexing` or against a matching PostgreSQL build. - Documentation historically lived at `lantern.dev/docs/develop/index`; the site has had outages (404s), so users should fall back to the in-repo `lantern_hnsw/sql/lantern.sql` and the `README.md` as the canonical reference until the hosted docs recover. ## Versioning and Compatibility Releases are tracked as semantic versions of the PostgreSQL extension. As of the latest tag (v0.5.0), Lantern ships: - Official PostgreSQL 17 support (added in #348). - Feature-gated embedding, autotune, and external indexing jobs (added in #349). - An updated dependency tree to keep pace with `usearch` and `pgvector` submodules used in tests. Operators upgrading across major PostgreSQL versions should rebuild both `lantern_hnsw/` and `lantern_cli/` and confirm that the GUCs controlling external indexing still match the daemon's listening address and SSL configuration. --- ## lantern_hnsw C Extension Internals ### Related Pages Related topics: [Lantern Overview and Component Architecture](#page-1), [lantern_cli and lantern_extras Tooling](#page-3), [Build, Installation, Release, and Known Failure Modes](#page-4)
Related Source Files The following source files were used to generate this page: - [lantern_hnsw/src/hnsw.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw.c) - [lantern_hnsw/src/hnsw.h](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw.h) - [lantern_hnsw/src/hnsw/build.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/build.c) - [lantern_hnsw/src/hnsw/insert.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/insert.c) - [lantern_hnsw/src/hnsw/delete.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/delete.c) - [lantern_hnsw/src/hnsw/scan.c](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/hnsw/scan.c) - [lantern_hnsw/src/CMakeLists.txt](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/src/CMakeLists.txt)
# lantern_hnsw C Extension Internals The `lantern_hnsw` directory hosts the compiled PostgreSQL extension that implements the lantern vector index access method. It is the low-level layer behind the user-facing `lantern` extension: SQL operators, index AM hooks, and type-casts all flow into this C code. The build is a CMake project configured by `lantern_hnsw/src/CMakeLists.txt`, and the resulting shared library is loaded by Postgres as a standard `PG_MODULE_MAGIC` extension. Source: [lantern_hnsw/src/hnsw.c:1-1]() and [lantern_hnsw/src/CMakeLists.txt:1-1]() ## Role in the Repository `lantern_hnsw` is one half of the project's split architecture. The `lantern` SQL extension wires up operators (`<->`, distance functions, type I/O) and supports background jobs; `lantern_hnsw` provides the actual index access method, the HNSW graph data structure, and persistence to disk. Everything in `lantern_hnsw/src/hnsw/` is paired with `lantern_hnsw/src/hnsw.h`, which holds the shared in-memory layout for the index, the per-tuple `HnswTuple` structure, and the PostgreSQL index-handle wrapper (`HnswHandle`). Source: [lantern_hnsw/src/hnsw.h:1-1]() The module's responsibilities are tight: - Parse and validate the `USING lantern` operator-class options (`dim`, `dist_fn`, `ef_construction`, `ef`, `M`, etc.). - Manage a single in-memory HNSW graph per index, built lazily during the first `INSERT` and updated incrementally. - Serve kNN queries using the standard Postgres index AM callbacks (`ambuild`, `aminsert`, `ambeginscan`, `amgettuple`, `amrescan`, `amendscan`, `amvacuumcleanup`). - Marshal vectors between Postgres `Datum` representations and internal `float`/`lvector` storage. - Stream the finalized graph to a file so the index survives a server restart. Source: [lantern_hnsw/src/hnsw.c:1-1]() ## Build, Insert, and Maintenance Pipeline Index construction is split between an external build path (recommended for large datasets) and the incremental build path driven by `INSERT`s. 1. **Access-method registration** happens in `hnsw.c`, where the index AM is wired into Postgres' `IndexAmRoutine`. Source: [lantern_hnsw/src/hnsw.c:1-1]() 2. **Index creation** parses operator-class options, allocates the on-disk metadata page (storing `dim`, element type, and a magic/version field), and registers the index for incremental population. Source: [lantern_hnsw/src/hnsw.h:1-1]() 3. **Incremental inserts** are handled in `insert.c`. Each new tuple is converted to an internal vector, a graph search finds its neighbors, and edges are bidirectionally linked. Writes hit Postgres' buffer manager and WAL so recovery is correct after a crash. Source: [lantern_hnsw/src/hnsw/insert.c:1-1]() 4. **Bulk builds** use `build.c`. When `lantern_create_index` is called in `external` mode the work is offloaded to a separate `lantern-cli` process over a TCP socket, then the resulting file is read back in chunks and persisted through `StoreExternalIndexNodes`. Source: [lantern_hnsw/src/hnsw/build.c:1-1]() 5. **Maintenance and deletes** are handled in `delete.c`. The routine calls into Postgres' `vacuum_delay_point()` to remain cooperative during long GC passes — this is the same point that breaks the build against PostgreSQL 18 beta 1, where the helper gained a mandatory `is_analyze` argument. Source: [lantern_hnsw/src/hnsw/delete.c:42-42]() ```mermaid flowchart LR SQL[SQL: CREATE INDEX / INSERT / SELECT] --> AM[Index AM in hnsw.c] AM --> INSERT[insert.c
incremental] AM --> BUILD[build.c
bulk via lantern-cli] AM --> SCAN[scan.c
kNN search] AM --> DEL[delete.c
vacuum cleanup] BUILD -- TCP socket --> CLI[lantern-cli external] CLI -- file chunks --> BUILD ``` A reusable 2000-dimensional limit is enforced at vector-parse time, which is why embeddings larger than 2000 elements (for example OpenAI `text-embedding-3-large` at 3072) are rejected up-front rather than failing inside the HNSW graph. Source: [lantern_hnsw/src/hnsw.h:1-1]() ## Query Path and Streaming Selects For `SELECT ... ORDER BY vec <-> $1 LIMIT k`, Postgres calls `ambeginscan`, optionally `amrescan` for parameterized rescans, and then `amgettuple` per row. The streaming variant introduced in 0.3.1 (PR #322) pushes a custom executor node so results can flow without materializing the full candidate list. Source: [lantern_hnsw/src/hnsw/scan.c:1-1]() Internally the search: - Loads the index from the shared buffer if it is not already resident, attaching it to the backend's `HnswHandle`. - Walks from the graph's entry point down through the upper layers using the configured `ef_construction` beam width, then refines the candidate set at layer 0 with `ef`. - Streams neighbors into a binary heap of size `k`, applying the `<->` distance function specified by the operator class. Source: [lantern_hnsw/src/hnsw/scan.c:1-1]() When the extension is built in `external` mode, the index server reports connection lines such as `[Lantern External Index] New connection: 127.0.0.1:63501` and may surface lower-level errno failures (for example `os error 22` on macOS when the index file descriptor is closed before the socket stream is fully drained). Source: [lantern_hnsw/src/hnsw/build.c:1-1]() ## Version Compatibility and Operational Notes The extension is tested against the PostgreSQL versions enumerated in the CMake configuration and against the release matrix published in the changelog. The 0.5.0 release notes confirm explicit `pg17` support and a feature gate that hides embedding, autotune, and external-indexing jobs from the daemon when not needed. Source: [lantern_hnsw/src/CMakeLists.txt:1-1]() Two outstanding community-reported failure modes are rooted in `lantern_hnsw` internals rather than the SQL wrapper: - A PostgreSQL 18 beta 1 build failure caused by a changed `vacuum_delay_point()` signature inside `delete.c` (issue #375). - A macOS `EINVAL` (`os error 22`) when an external index's file descriptor is closed before the streamed buffer is fully consumed (issue #370). Knowing which C file owns each operation makes it much easier to triage these reports, because reproducer logs almost always point to either the access-method dispatcher in `hnsw.c`, the incremental path in `insert.c`, the bulk/external path in `build.c`, or the maintenance path in `delete.c`. --- ## lantern_cli and lantern_extras Tooling ### Related Pages Related topics: [Lantern Overview and Component Architecture](#page-1), [lantern_hnsw C Extension Internals](#page-2), [Build, Installation, Release, and Known Failure Modes](#page-4)
Related Source Files The following source files were used to generate this page: - [lantern_cli/src/main.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/main.rs) - [lantern_cli/src/cli.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/cli.rs) - [lantern_cli/src/types.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/types.rs) - [lantern_cli/src/daemon/mod.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/mod.rs) - [lantern_cli/src/daemon/cli.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/cli.rs) - [lantern_cli/src/daemon/autotune_jobs.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/autotune_jobs.rs) - [lantern_cli/Cargo.toml](https://github.com/lanterndata/lantern/blob/main/lantern_cli/Cargo.toml) - [lantern_cli/src/daemon/embeddings_jobs.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/embeddings_jobs.rs) - [lantern_cli/src/daemon/external_indexing_jobs.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/external_indexing_jobs.rs) - [lantern_cli/src/daemon/http_client.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/http_client.rs) - [lantern_cli/src/daemon/db.rs](https://github.com/lanterndata/lantern/blob/main/lantern_cli/src/daemon/db.rs) - [lantern_extras/CMakeLists.txt](https://github.com/lanterndata/lantern/blob/main/lantern_extras/CMakeLists.txt) - [lantern_extras/src/daemon_tools/daemon_tools.c](https://github.com/lanterndata/lantern/blob/main/lantern_extras/src/daemon_tools/daemon_tools.c)
# lantern_cli and lantern_extras Tooling ## Overview and Scope The `lantern_cli` is a standalone Rust binary that complements the core `lantern` PostgreSQL extension. While the C-based extension provides the HNSW index implementation and SQL types, `lantern_cli` operates as an out-of-process companion daemon and client that manages long-running background tasks. The companion library `lantern_extras` (a PostgreSQL extension written in C) bridges SQL-level operations to the daemon via background workers and helper functions. `lantern_cli` exposes a binary entry point through `main.rs`, which dispatches between direct CLI commands and a long-lived daemon mode. Source: [lantern_cli/src/main.rs:1-50](). The release history shows the tooling has matured quickly: - v0.3.2 introduced external indexing via TCP socket, plus SSL support ([v0.3.2 release notes](#)). - v0.3.4 added an external index router and refactored `StoreExternalIndexNodes` ([v0.3.4 release notes](#)). - v0.4.0 merged `lantern_extras` into the main repository and shipped macOS binaries for `lantern_cli` ([v0.4.0 release notes](#)). - v0.4.1 bumped `lantern_cli` to 0.4.0 ([v0.4.1 release notes](#)). - v0.5.0 added PostgreSQL 17 support and feature-gated embedding, autotune, and external-indexing daemon jobs ([v0.5.0 release notes](#)). ## Architecture and Components The tooling is split across three logical layers. ### 1. lantern_cli CLI binary (Rust) `main.rs` parses arguments and routes to either an immediate command or the daemon subsystem. Source: [lantern_cli/src/main.rs:1-50](). The CLI argument definitions live in `cli.rs`, while shared structures such as connection settings, index parameters, and job descriptors are centralized in `types.rs` to keep both modes consistent. Source: [lantern_cli/src/cli.rs:1-40](), [lantern_cli/src/types.rs:1-40](). The `Cargo.toml` declares the package metadata and feature flags that gate daemon subsystems. Source: [lantern_cli/Cargo.toml:1-40](). ### 2. lantern_cli daemon subsystem The daemon lives under `lantern_cli/src/daemon/`. It is composed of: - `mod.rs` — entry point that initializes logging, connects to PostgreSQL, and starts the worker loops. Source: [lantern_cli/src/daemon/mod.rs:1-60](). - `cli.rs` — daemon-specific subcommands (start, stop, status). Source: [lantern_cli/src/daemon/cli.rs:1-50](). - `db.rs` — PostgreSQL connection pool and schema introspection helpers. Source: [lantern_cli/src/daemon/db.rs:1-50](). - `http_client.rs` — outbound HTTP client used to call embedding providers. Source: [lantern_cli/src/daemon/http_client.rs:1-40](). - `autotune_jobs.rs` — worker that picks parameter-tuning jobs from the queue and runs HNSW parameter experiments against the target table. Source: [lantern_cli/src/daemon/autotune_jobs.rs:1-50](). - `embeddings_jobs.rs` — worker that streams rows to an embedding service and writes vectors back. Source: [lantern_cli/src/daemon/embeddings_jobs.rs:1-50](). - `external_indexing_jobs.rs` — worker that builds an HNSW index out-of-process and streams the binary index file to the PostgreSQL backend over a TCP socket. Source: [lantern_cli/src/daemon/external_indexing_jobs.rs:1-50](). ### 3. lantern_extras PostgreSQL extension `lantern_extras` is a SQL/C extension that registers background worker types, helper SQL functions, and the `lantern_extras` schema used to queue jobs. It is built via its own `CMakeLists.txt` and exposes `daemon_tools.c` which contains C glue code that PostgreSQL background workers call to hand work off to the daemon. Source: [lantern_extras/CMakeLists.txt:1-30](), [lantern_extras/src/daemon_tools/daemon_tools.c:1-50](). ## Data Flow and Job Lifecycle The diagram below traces a typical external-indexing request from SQL to the Rust daemon and back. ```mermaid sequenceDiagram participant SQL as PostgreSQL (lantern_extras) participant BG as Background Worker participant Daemon as lantern_cli daemon participant Net as TCP/SSL Socket SQL->>BG: CREATE INDEX ... USING lantern_extras_external BG->>Daemon: enqueue job row Daemon->>Daemon: build HNSW in Rust Daemon->>Net: stream index bytes (+ header) Net->>BG: receive chunked file BG->>SQL: StoreExternalIndexNodes() SQL-->>SQL: index visible to planner ``` Key control points referenced by the sources: - The job queue schema and `StoreExternalIndexNodes` plumbing live in `lantern_extras/src/daemon_tools/daemon_tools.c`. Source: [lantern_extras/src/daemon_tools/daemon_tools.c:1-80](). - The external indexing worker negotiates the connection and reads the header (vector element bits, file size) before streaming the index. Source: [lantern_cli/src/daemon/external_indexing_jobs.rs:1-60](). - Feature flags in `Cargo.toml` allow builds to disable the embedding, autotune, or external-indexing workers when not needed. Source: [lantern_cli/Cargo.toml:1-40](). ## Feature Gating, Platforms, and Known Limitations As of v0.5.0, the embedding, autotune, and external-indexing workers are gated behind Cargo features so that minimal builds only include the daemon core. Source: [lantern_cli/Cargo.toml:1-40](), [v0.5.0 release notes](#). Platform support: - macOS binaries are published for `lantern_cli` starting in v0.4.0. Source: [v0.4.0 release notes](#). - PostgreSQL 17 is officially supported in v0.5.0. Source: [v0.5.0 release notes](#). - A PostgreSQL 18 beta 1 build failure was reported in issue #375, caused by `vacuum_delay_point()` requiring the new `is_analyze` argument. Source: [issue #375](#). - A macOS build issue was tracked in #352, where local compilation from source produced errors after `make`. Source: [issue #352](#). - An `os error 22` was raised on macOS 15.2 with PostgreSQL 16 from Homebrew when an external index was built against `lantern_hnsw` master (issue #370). Source: [issue #370](#). - The project currently caps vector dimensions at 2000 (issue #374), which limits external-indexing payloads the daemon can stream. Source: [issue #374](#). - Earlier community-reported documentation 404s on lantern.dev (issue #371) have since been marked closed. Source: [issue #371](#). ## Operational Notes When deploying `lantern_cli`: 1. Build the Rust binary with the desired feature set: `cargo build --release --features "embeddings,autotune,external-indexing"`. Source: [lantern_cli/Cargo.toml:1-40](). 2. Install `lantern_extras` into the target PostgreSQL instance so the background workers and helper functions are registered. Source: [lantern_extras/CMakeLists.txt:1-30](). 3. Start the daemon with `lantern_cli daemon start`; it reads connection settings from CLI flags or environment variables defined in `types.rs`. Source: [lantern_cli/src/daemon/cli.rs:1-50](), [lantern_cli/src/types.rs:1-40](). 4. SQL-level operations triggered inside PostgreSQL enqueue rows that the daemon polls via the helpers in `daemon_tools.c`. Source: [lantern_extras/src/daemon_tools/daemon_tools.c:1-80](). This split — extension for in-database primitives, daemon for heavy/long-running work, and `lantern_extras` for the glue — is the recurring pattern across recent releases and is what allows Lantern to build HNSW indexes externally without blocking the database process. --- ## Build, Installation, Release, and Known Failure Modes ### Related Pages Related topics: [Lantern Overview and Component Architecture](#page-1), [lantern_hnsw C Extension Internals](#page-2), [lantern_cli and lantern_extras Tooling](#page-3)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/lanterndata/lantern/blob/main/README.md) - [CHANGELOG.md](https://github.com/lanterndata/lantern/blob/main/CHANGELOG.md) - [CONTRIBUTING.md](https://github.com/lanterndata/lantern/blob/main/CONTRIBUTING.md) - [.dockerignore](https://github.com/lanterndata/lantern/blob/main/.dockerignore) - [docker/Dockerfile](https://github.com/lanterndata/lantern/blob/main/docker/Dockerfile) - [docker/Dockerfile.cli](https://github.com/lanterndata/lantern/blob/main/docker/Dockerfile.cli) - [lantern_hnsw/CMakeLists.txt](https://github.com/lanterndata/lantern/blob/main/lantern_hnsw/CMakeLists.txt)
# Build, Installation, Release, and Known Failure Modes ## Overview Lantern is a PostgreSQL extension that implements the HNSW (Hierarchical Navigable Small World) algorithm for vector similarity search. Building, packaging, and releasing the project requires coordinating three components: the PostgreSQL extension C code under `lantern_hnsw/`, the companion `lantern_extras` and `lantern_cli` packages, and the optional standalone external index daemon. The project supports local source builds, Docker-based builds, and packaged releases distributed via GitHub. Source: [README.md:1-40]() This page documents the build pipeline, installation entry points, release cadence, and the failure modes that the community has reported (such as PostgreSQL 18 beta incompatibilities, macOS build errors, and runtime dimension limits). ## Build Process ### Local Source Build The repository is a CMake project rooted at `lantern_hnsw/CMakeLists.txt`. The canonical local build sequence is: ```bash git clone --recursive https://github.com/lanterndata/lantern.git cd lantern cmake -DMARCH_NATIVE=ON -S lantern_hnsw -B build make -C build install -j ``` `MARCH_NATIVE=ON` enables CPU-specific vector instruction optimizations. The `--recursive` flag is required because several dependencies (notably pgvector) live in git submodules. Source: [lantern_hnsw/CMakeLists.txt:1-40]() ### Docker Build Two Dockerfiles are provided: - `docker/Dockerfile` builds the PostgreSQL extension and ships a runnable image that wires the extension into a Postgres data directory. - `docker/Dockerfile.cli` builds the `lantern_cli` standalone binary, mirroring the macOS and Linux CLI distribution targets added in v0.4.0. The `.dockerignore` file excludes transient build artifacts (e.g., the `build/` directory) and local git metadata so the build context stays small. Source: [.dockerignore:1-30](), [docker/Dockerfile:1-60](), [docker/Dockerfile.cli:1-40]() ### Contributing and CI The `CONTRIBUTING.md` file documents the developer workflow, coding style for C and SQL, the test harness, and the matrix of PostgreSQL versions exercised in CI. As of the v0.5.0 release, pg17 support was added officially, broadening the supported matrix. Source: [CONTRIBUTING.md:1-80]() ## Installation Once the extension is compiled, installation follows the standard PostgreSQL extension pattern: 1. Place the shared object into the PostgreSQL `lib` directory (handled automatically by `make install`). 2. Add `lantern` to `shared_preload_libraries` if using background workers or the external index daemon. 3. In the target database, run `CREATE EXTENSION lantern;` to load the SQL bindings. The `lantern_cli` binary is distributed as a separate artifact (Linux and macOS binaries are produced from `docker/Dockerfile.cli`). It is used to build HNSW indexes out-of-process and stream them back to PostgreSQL over a TCP socket, a feature introduced in v0.3.2 and stabilized across v0.3.3 and v0.3.4. Source: [CHANGELOG.md:1-80]() ## Release History Lantern follows an iterative release model. The most recent releases are summarized in the table below. | Version | Headline Changes | |---|---| | v0.5.0 | pg17 support, dependency updates, feature-gated daemon jobs, `lantern_extras` coverage via grcov | | v0.4.1 | `lantern_cli` bumped to 0.4.0, public `SELECT` grant on job tables | | v0.4.0 | Fix pgvector installation in CI via submodules, move `lantern_extras` into the lantern repo, ship macOS `lantern_cli` binaries | | v0.3.4 | Refactored `StoreExternalIndexNodes` to receive index file in chunks, external index router, element-bits in header | | v0.3.3 | Close index FD only when not built via socket | | v0.3.2 | External indexing via TCP socket, SSL support, extension version check refactor | | v0.3.1 | Streaming HNSW index selects | | v0.3.0 | Eliminate blockmap, single WAL pass for HNSW build | | v0.2.7 | HNSW delete handling | | v0.2.6 | Async task job-id fix | Source: [CHANGELOG.md:1-200]() The pattern is consistent: each minor bump corresponds to a single architectural change (delete handling → blockmap removal → single WAL pass → socket-based external indexing → version check refactor → pg17 support), with patch releases (v0.4.1, v0.3.3) hardening the prior minor. ## Known Failure Modes The community has reported several concrete failure patterns that anyone building or running Lantern should be aware of. ### PostgreSQL 18 beta 1 build failure On PostgreSQL 18 beta 1, the source fails to compile because the upstream signature of `vacuum_delay_point()` changed to require an `is_analyze` argument. The error originates in `lantern_hnsw/src/hnsw/delete.c:42` and breaks the build on the new server header. This is tracked in issue #375 and is a forward-compatibility gap, not a regression in Lantern's own logic. Source: issue #375 ### macOS local build failure Users building from `master` on macOS 15.2 (with Homebrew PostgreSQL 16) have reported build errors during the `make install` step. The failure is associated with the same `MARCH_NATIVE=ON` and external index code paths, suggesting that the combination of Apple Clang and the new socket-based external indexer (introduced in v0.3.2) is not yet fully stable on macOS. Track via issue #352. Source: issue #352 ### Vector dimension ceiling (2000) Lantern currently rejects vector dimensions greater than 2000 with the message `vector dimension 3072 is too large. LanternDB currently supports up to 2000dim vectors`. This blocks users of newer OpenAI embedding models such as `text-embedding-3-large` (3072 dims) without a manual recompile against a raised `LANTERN_MAX_DIM`. Tracked in issue #374. Source: issue #374 ### External index "os error 22" When building an external index on macOS 15.2, the daemon logs `[Lantern External Index] New connection: 127.0.0.1:63501` and then fails with an `os error 22` (EINVAL). This typically indicates a malformed or truncated header frame being sent to the socket; v0.3.4 added explicit element-bit headers to mitigate this class of error. Tracked in issue #370. Source: issue #370 ### Documentation 404s (transient) The `lantern.dev` documentation site has intermittently returned 404 for the `/docs/develop/*` paths. The issue was closed once the site was restored, but it is a reminder that release notes and the GitHub README are the canonical source of truth when the docs site is unavailable. Tracked in issue #371. Source: issue #371 ## Workflow Summary The end-to-end path from source to running extension is: 1. Clone with `--recursive` to fetch pgvector and other submodules. 2. Configure with CMake (optionally `MARCH_NATIVE=ON`). 3. Build and install with `make -C build install -j`. 4. `CREATE EXTENSION lantern;` in the target database. 5. Optionally install `lantern_cli` and start the external index daemon for out-of-process index construction. When hitting a build error, first check the PostgreSQL major version against the supported matrix (pg13–pg17 in v0.5.0; pg18 is not yet supported), then confirm the submodule state, and finally reproduce inside the Docker images as a clean baseline. Source: [README.md:20-80](), [docker/Dockerfile:1-60]() --- --- ## Pitfall Log Project: lanterndata/lantern Summary: Found 13 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification. ## 1. Installation risk - Installation risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/lanterndata/lantern/issues/370 ## 2. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: runtime_trace - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Repro command: `docker run --pull=always --rm -p 5432:5432 -e "POSTGRES_USER=$USER" -e "POSTGRES_PASSWORD=postgres" -v ./lantern_data:/var/lib/postgresql/data lanterndata/lantern:latest-pg15` - Evidence: identity.distribution | https://github.com/lanterndata/lantern ## 3. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/lanterndata/lantern/issues/375 ## 4. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/lanterndata/lantern/issues/352 ## 5. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/lanterndata/lantern ## 6. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/lanterndata/lantern/issues/371 ## 7. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/lanterndata/lantern/issues/374 ## 8. Runtime risk - Runtime risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: packet_text.keyword_scan | https://github.com/lanterndata/lantern ## 9. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/lanterndata/lantern ## 10. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/lanterndata/lantern ## 11. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/lanterndata/lantern ## 12. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/lanterndata/lantern ## 13. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/lanterndata/lantern