# lantern - Doramagic AI Context Pack

> Positioning: a pre-install experience and judgment asset. It helps the host AI get off to a good start, but it does not mean the project has already been installed, run, or validated.

## Sufficiency Principle

- **Sufficiency over compression**: The AI Context Pack should be sufficient for the host AI to understand the project's value, capability boundaries, entrypoints, risks, and evidence sources before starting work; it may be layered, but it does not aim for the shortest possible summary.
- **Compression policy**: Compress only noise and duplication, never context that affects judgment or the quality of the work.

## How the Host AI Should Use This

You are reading the AI Context Pack that Doramagic compiled for lantern. Treat it as pre-work context: help the user understand who it fits, what it can do, how to start, what must be verified after install, and where the risks are. Do not claim that you have already installed, run, or executed the target project.

## Claim Consumption Rules

- **Fact source**: Repo Evidence + Claim/Evidence Graph; the Human Wiki only supplies salience, terminology, and narrative structure.
- **Minimum status for a fact**: `supported`
- `supported`: May be used as a project fact, but the answer must cite the claim_id and evidence path.
- `weak`: Usable only as a low-confidence lead; the user must be asked to keep verifying.
- `inferred`: Usable only for risk notes or open questions; must not be packaged as a project fact.
- `unverified`: Must not be used as fact; state clearly that evidence is insufficient.
- `contradicted`: Must show the conflicting sources and must not force a single version on the user's behalf.

## Who It Fits Best

- **Users who want to understand an open-source project's value and boundaries before installing**: Current evidence comes mainly from project documentation. Claim: `clm_0002` unverified 0.25

## What It Can Do

- **Command-Line Startup or Install Flow** (Verify after install): The project documentation contains runnable commands; real use requires running them in a local or host environment. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al. Claim: `clm_0001` supported 0.86

## How to Start

- `git clone https://git.postgresql.org/git/postgresql.git` Evidence: `CONTRIBUTING.md` Claim: `clm_0003` supported 0.86
- `git clone --single-branch --branch REL_15_STABLE https://git.postgresql.org/git/postgresql.git --depth=1` Evidence: `CONTRIBUTING.md` Claim: `clm_0004` supported 0.86
- `git clone https://github.com/citusdata/pg_cron.git` Evidence: `ci/scripts/build-linux.sh` Claim: `clm_0005` unverified 0.25
- `git clone https://github.com/lanterndata/benchmark` Evidence: `ci/scripts/run-benchmarks.sh` Claim: `clm_0006` unverified 0.25
- `pip install -r core/requirements.txt` Evidence: `ci/scripts/run-benchmarks.sh` Claim: `clm_0007` unverified 0.25
- `pip install -r external/requirements.txt` Evidence: `ci/scripts/run-benchmarks.sh` Claim: `clm_0008` unverified 0.25
- `curl -k -o /tmp/rustup.sh https://sh.rustup.rs` Evidence: `ci/scripts/utils.sh` Claim: `clm_0009` unverified 0.25
- `git clone --recursive https://github.com/lanterndata/lantern.git -b $BRANCH` Evidence: `ci/scripts/utils.sh` Claim: `clm_0010` unverified 0.25
- `git clone --recursive https://github.com/lanterndata/pgvector.git -b "v${PGVECTOR_VERSION}"` Evidence: `ci/scripts/utils.sh` Claim: `clm_0011` unverified 0.25
- `pip install -r /tmp/lantern/lantern_hnsw/scripts/requirements.txt` Evidence: `ci/scripts/utils.sh` Claim: `clm_0012` unverified 0.25

## Continue-or-Stop Decision Card

- **Current recommendation**: Sandbox trial only
- **Why**: The project has signals of install commands, host configuration, or local writes; do not go straight into your primary environment—trial it in isolation first.

### 30-Second Read

- **What to do now**: Sandbox trial only
- **Minimum safe next step**: Run Prompt Preview first; if you still want to install, trial only in an isolated environment
- **Do not trust yet**: Real output quality cannot be trusted before install.
- **Continuing will touch**: Command execution, Local environment or project files, Host AI context

### What You Can Trust Now

- **Capability exists: Command-Line Startup or Install Flow** (supported): You can trust that the project contains signals of this capability; whether it fits your specific task still needs trial or after-install verification. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al. Claim: `clm_0001` supported 0.86
- **There are Quick Start / install-command signals** (supported): You can trust that the docs mention a startup or install entrypoint; do not run it directly in your primary environment because of that. Evidence: `CONTRIBUTING.md` Claim: `clm_0003` supported 0.86

### What You Cannot Trust Yet

- **Real output quality cannot be trusted before install.** (unverified): Prompt Preview can only show how it guides you; it cannot prove result quality in the real project.
- **Host AI version compatibility cannot be trusted before install.** (unverified): Host loading rules and version differences across Claude, Cursor, Codex, Gemini, and others must be verified in a real environment.
- **That it will not pollute your existing host AI's behavior cannot be trusted directly.** (inferred): Skill, plugin, and AGENTS/CLAUDE/GEMINI instructions may change the host AI's default behavior.
- **Safe rollback cannot be assumed by default.** (unverified): Unless the project clearly provides uninstall and recovery instructions, verify in an isolated environment first.
- **After a real install, is it compatible with the user's current host AI version?** (unverified): Compatibility can only be verified in the actual host environment.
- **Does the project's output quality meet the user's specific task?** (unverified): The pre-install preview can only show flow and boundaries; it cannot replace real evaluation.
- **Do the install commands require network access, permissions, or global writes?** (unverified): This affects install risk in both enterprise and personal environments. Evidence: `CONTRIBUTING.md`

### What Continuing Will Touch

- **Command execution**: Package managers, network downloads, the local plugin directory, project config, or the user's home directory. Why: Running the very first command can already change your environment; decide whether it is worth running first. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al.
- **Local environment or project files**: Install results, plugin caches, project config, or local dependency directories. Why: The write scope and rollback path cannot be proven before install and need isolated verification. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al.
- **Host AI context**: The AI Context Pack, Prompt Preview, Skill routing, risk rules, and project facts. Why: Importing context affects the host AI's later judgment, so avoid packaging unverified items as facts.

### Minimum Safe Next Steps

- **Run Prompt Preview first**: Use a pre-install interactive trial to judge whether the way of working fits; it needs no authorization or environment change. (applies when: Applies to any project, especially when output quality is unknown.)
- **Trial-install only in an isolated directory or a test account**: Avoid letting install commands pollute your primary host AI, real projects, or home directory. (applies when: When there are signals of command execution, plugin config, or local writes.)
- **After install, verify just one minimal task**: Verify loading, compatibility, output quality, and rollback first, then decide whether to use it deeply. (applies when: When moving from a trial into a real workflow.)

### Exit Plan

- **Preserve the pre-install state**: Record the original host config and project state so you can later judge whether it is recoverable.
- **Record the install commands and written paths**: Without clear uninstall instructions, you at least need to know which directories or configs to clean up manually.
- **If there is no rollback path, do not enter your primary environment**: No rollback is a blocker before continuing; do not proceed on trust or luck.

## What Can Only Be Previewed

- Explain who the project fits and what it can do
- Demonstrate a typical conversation flow based on project docs
- Help the user decide whether it is worth installing or researching further

## What Must Be Verified After Install

- Actually installing the Skill, plugin, or CLI
- Running scripts, modifying local files, or accessing external services
- Verifying real output quality, performance, and compatibility

## Boundary & Risk Decision Card

- **Mistaking the pre-install preview for a real run**: The user may overestimate how much configuration, permission, and compatibility verification the project has already done. Mitigation: Clearly separate prompt_preview_can_do from runtime_required. Claim: `clm_0024` inferred 0.45
- **Command execution will modify the local environment**: Install commands may write to the user's home directory, the host plugin directory, or project configuration. Mitigation: Run in an isolated environment or a test account first. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al. Claim: `clm_0025` supported 0.86
- **To confirm**: After a real install, is it compatible with the user's current host AI version?. Why: Compatibility can only be verified in the actual host environment.
- **To confirm**: Does the project's output quality meet the user's specific task?. Why: The pre-install preview can only show flow and boundaries; it cannot replace real evaluation.
- **To confirm**: Do the install commands require network access, permissions, or global writes?. Why: This affects install risk in both enterprise and personal environments.

## Pre-Work Working Context

### Loading Order

- First read how_to_use.host_ai_instruction to establish the boundaries of this pre-install judgment asset.
- Read claim_graph_summary to confirm facts come from the Claim/Evidence Graph, not the Human Wiki narrative.
- Then read intended_users, capabilities, and quick_start_candidates to judge whether the user is a match.
- When you need to carry out a concrete task, check role_skill_index first, then evidence_index.
- For real install, file modification, network access, performance, or compatibility questions, turn to risk_card and boundaries.runtime_required.

### Task Routes

- **Command-Line Startup or Install Flow**: State that this is an after-install capability first, then give a pre-install checklist. Boundary: Must be verified after a real install or run. Evidence: `CONTRIBUTING.md`, `ci/scripts/build-linux.sh`, `ci/scripts/run-benchmarks.sh`, `ci/scripts/utils.sh` et al. Claim: `clm_0001` supported 0.86

### Context Scale

- Total files: 288
- Important-file coverage: 40/288
- Evidence index entries: 55
- Role / Skill entries: 5

### Handling Insufficient Evidence

- **missing_evidence**: State that evidence is insufficient and ask the user for the target file, a README section, or after-install verification records; do not fill in facts.
- **out_of_scope_request**: State that the task is beyond the current AI Context Pack's evidence scope and suggest the user check the Human Manual or verify after a real install.
- **runtime_request**: Provide a pre-install checklist and command sources, but do not run commands for the user or claim they have been run.
- **source_conflict**: Show the conflicting sources side by side, mark them as unverified, and do not force a single version.

## Prompt Recipes

### Fit assessment

- Goal: Judge whether this project fits the user's current task.
- Expected output: A fit conclusion, key reasons, evidence citations, what can be previewed before install, what must be verified after install, and a next-step recommendation.

```text
Based on the AI Context Pack for lantern, ask me 3 necessary questions first, then judge whether it fits my task. The answer must cover: who it fits, what it can do, what it cannot do, whether it is worth installing, and where the evidence comes from. Every project fact must cite evidence_refs, source_paths, or a claim_id.
```

### Pre-install experience

- Goal: Let the user feel the core workflow before installing, while avoiding packaging the preview as real capability or a marketing promise.
- Expected output: An experience script with boundary labels, an after-install verification checklist, and a cautious recommendation; with no real-run promises or strong marketing language.

```text
Treat lantern as a pre-install experience asset, not an already-installed tool or a real runtime environment.

Output exactly four parts:
1. Ask me 3 necessary questions first.
2. Give an "experience script": use the three labels [Previewable before install], [Must verify after install], and [Insufficient evidence] to show how it might guide the workflow.
3. Give an after-install verification checklist: list which capabilities can only be confirmed after a real install, real host loading, and a real project run.
4. Give a cautious recommendation: only "worth researching/trialing further", "add information before deciding", or "not recommended to continue"; do not endorse the project.

Hard boundaries:
- Do not claim you have installed, run, executed tests, modified files, or produced real results.
- Do not write promise-like phrasing such as "auto-adapts", "guarantees passing", "perfect fit", or "strongly recommend installing".
- If you describe how it works after install, you must use a conditional such as "if installed successfully and the host loads the Skill correctly, it might...".
- The experience script may only be written as "example lines / hypothetical flow": use "might ask / might suggest / might show", not "has written, has generated, has passed, is running, is generating".
- Prompt Preview does not hand out install commands; if the user is ready to trial, only prompt them to read Quick Start and the Risk Card first and to verify in an isolated environment.
- Every project fact must come from a supported claim, evidence_refs, or source_paths; inferred/unverified items can only be risks or open questions.

```

### Role / Skill selection

- Goal: Pick the best-matching asset from the project's roles or Skills.
- Expected output: A list of candidate roles or Skills, each with an applicable scenario, evidence paths, risk boundary, and whether after-install verification is needed.

```text
Read role_skill_index and recommend 3-5 of the most relevant roles or Skills for my target task. For each recommendation, state the applicable scenario, likely output, risk boundary, and evidence_refs.
```

### Risk pre-check

- Goal: Identify environment, permission, rule-conflict, and quality risks before installing or adopting.
- Expected output: A checklist of environment, permission, dependency, license, host-conflict, quality risk, and unknown items.

```text
Based on risk_card, boundaries, and quick_start_candidates, give me a pre-install risk pre-check list. Do not run commands for me; only explain what I should check, why, and what impact a failure would have.
```

### Host AI kickoff instruction

- Goal: Turn the project context into a host AI instruction for the start of a conversation.
- Expected output: A pre-work instruction with clear boundaries and clear evidence citations, suitable to copy to a host AI.

```text
Based on the AI Context Pack for lantern, generate a pre-work instruction I can paste to my host AI. This instruction must obey not_runtime=true and must not claim the project has been installed, run, or produced real results.
```

## Role / Skill Index

- Indexed 5 role / Skill / project-doc entries.

- **Lantern Extras** (project_doc): This extension makes it easy to experiment with embeddings from inside a Postgres database. We use this extension along with Lantern https://github.com/lanterndata/lantern to make vector operations performant. But all the helpers here are standalone and may be used without the main database. Activation hint: Reference this when the user needs to understand the project's structure, install path, or boundaries. Evidence: `lantern_extras/README.md`
- **Suppressions for Clang Sanitizers** (project_doc): This folder contains supression files https://clang.llvm.org/docs/SanitizerSpecialCaseList.html for running lantern using Clang's AddressSanitizer https://clang.llvm.org/docs/AddressSanitizer.html and UndefinedBehaviorSanitizer https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html , taken from timescale's regression suite. Activation hint: Reference this when the user needs to understand the project's structure, install path, or boundaries. Evidence: `lantern_hnsw/scripts/sanitizers/suppressions/README.md`
- **Running tests** (project_doc): Thanks for considering contributing! The information below is intended to help you contribute. Activation hint: Reference this when the user needs to understand the project's structure, install path, or boundaries. Evidence: `CONTRIBUTING.md`
- **Overview** (project_doc): This is a ruby program that uses postgres internal catalog table invariants to exhaustively verify correctness of Postgres database extension update scripts. Activation hint: Reference this when the user needs to understand the project's structure, install path, or boundaries. Evidence: `lantern_hnsw/scripts/test_updates/README.md`
- **CHANGELOG** (project_doc):  Activation hint: Reference this when the user needs to understand the project's structure, install path, or boundaries. Evidence: `CHANGELOG.md`

## Evidence Index

- Indexed 55 evidence entries.

- **Lantern Extras** (documentation): This extension makes it easy to experiment with embeddings from inside a Postgres database. We use this extension along with Lantern https://github.com/lanterndata/lantern to make vector operations performant. But all the helpers here are standalone and may be used without the main database. Evidence: `lantern_extras/README.md`
- **Suppressions for Clang Sanitizers** (documentation): This folder contains supression files https://clang.llvm.org/docs/SanitizerSpecialCaseList.html for running lantern using Clang's AddressSanitizer https://clang.llvm.org/docs/AddressSanitizer.html and UndefinedBehaviorSanitizer https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html , taken from timescale's regression suite. Evidence: `lantern_hnsw/scripts/sanitizers/suppressions/README.md`
- **Running tests** (documentation): Thanks for considering contributing! The information below is intended to help you contribute. Evidence: `CONTRIBUTING.md`
- **Overview** (documentation): This is a ruby program that uses postgres internal catalog table invariants to exhaustively verify correctness of Postgres database extension update scripts. Evidence: `lantern_hnsw/scripts/test_updates/README.md`
- **License** (source_file): GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007 Evidence: `LICENSE`
- **CHANGELOG** (documentation): CHANGELOG Evidence: `CHANGELOG.md`
- **.dockerignore** (source_file): .git .cargo .github target .DS Store lantern hnsw/build Evidence: `.dockerignore`
- **Cargo** (source_file): members = "lantern extras", "lantern cli" Evidence: `Cargo.toml`
- **Dockerfile** (source_file): FROM postgres:$PG VERSION-bookworm ARG PG VERSION Evidence: `docker/Dockerfile`
- **Copy the files in your machine to the Docker image** (source_file): FROM rust:1.82.0 as build Copy the files in your machine to the Docker image WORKDIR /app COPY lantern cli . ENV ORT STRATEGY="system" ARG RUSTFLAGS="" ENV RUSTFLAGS=$RUSTFLAGS ARG CXXFLAGS="" ENV CXXFLAGS=$CXXFLAGS Evidence: `docker/Dockerfile.cli`
- **Copy the files in your machine to the Docker image** (source_file): FROM rust:1.82.0 as build Copy the files in your machine to the Docker image WORKDIR /app COPY lantern cli . Evidence: `docker/Dockerfile.cli.cuda`
- **fix pg cron at the latest commit of the time** (source_file): ARG VERSION=15 ARG PGVECTOR VERSION=0.5.1 fix pg cron at the latest commit of the time ARG PG CRON COMMIT SHA=9490f9cc9803f75105f2f7d89839a998f011f8d8 Evidence: `docker/Dockerfile.dev`
- **To use GDB inside container run docker like this:** (source_file): To use GDB inside container run docker like this: docker build . -t pg-debug docker run --cap-add=SYS PTRACE -p 5433:5432 -d --name debug-pg pg-debug Then exec to it docker exec -ti -u root debug-bg bash gdb -p $pid of pg backend Evidence: `docker/Dockerfile.pg`
- **See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html** (source_file): package name = "lantern cli" version = "0.5.1" edition = "2021" Evidence: `lantern_cli/Cargo.toml`
- **Cargo** (source_file): package name = "lantern extras" version = "0.6.0" edition = "2021" Evidence: `lantern_extras/Cargo.toml`
- **us ROOT variables from inside find package calls** (source_file): cmake minimum required VERSION 3.3 include CheckSymbolExists Evidence: `lantern_hnsw/CMakeLists.txt`
- **Cli** (source_file): use lantern cli::daemon::cli::DaemonArgs; ⋮---- use lantern cli::external index::cli::IndexServerArgs; use lantern cli::http server::cli::HttpServerArgs; use lantern cli::index autotune::cli::IndexAutotuneArgs; use lantern cli::pq::cli::PQArgs; ⋮---- pub enum Commands { ⋮---- pub struct Cli { Evidence: `lantern_cli/src/cli.rs`
- **Autotune Jobs** (source_file): use crate::daemon::helpers::anyhow wrap connection; use crate::external index::cli::UMetricKind; use crate::index autotune::cli::IndexAutotuneArgs; use crate::logger::Logger; ⋮---- use crate::utils::get full table name; use std::collections::HashMap; use std::sync::Arc; use std::time::SystemTime; use tokio::sync::RwLock; ⋮---- use tokio util::sync::CancellationToken; ⋮---- pub struct AutotuneJob { ⋮---- impl AutotuneJob { pub fn new row: Row, db uri: &str - AutotuneJob { ⋮---- db uri: db uri.to owned , ⋮---- pub async fn autotune job processor ⋮---- Ok ⋮---- async fn autotune worker ⋮---- let jobs table name = Arc::new get full table name &schema, &table ; ⋮---- logger.info "Autotune worker… Evidence: `lantern_cli/src/daemon/autotune_jobs.rs`
- **Cli** (source_file): use crate::logger; ⋮---- pub enum LogLevel { ⋮---- impl LogLevel { pub fn value &self - logger::LogLevel { ⋮---- pub struct DaemonArgs { Evidence: `lantern_cli/src/daemon/cli.rs`
- **Mod** (source_file): pub mod autotune jobs; pub mod cli; ⋮---- mod client embedding jobs; ⋮---- pub mod embedding jobs; mod helpers; mod types; ⋮---- use futures::StreamExt; use std::collections::HashMap; use std::sync::Arc; ⋮---- use tokio util::sync::CancellationToken; ⋮---- use crate::types::AnyhowVoidResult; ⋮---- lazy static! { ⋮---- async fn get target databases ⋮---- if args.target db.is none && args.master db.is none { ⋮---- if let Some target db = args.target db.as ref { ⋮---- .iter .map db str - Result { Ok TargetDB::from uri db str ? } ⋮---- let db uri = args.master db.as ref .unwrap ; ⋮---- tokio::spawn async move { connection.await.unwrap } ; ⋮---- Ok client .query &format! ⋮---- .filter map ⋮----… Evidence: `lantern_cli/src/daemon/mod.rs`
- **Cli** (source_file): pub use super::core::Runtime; ⋮---- pub enum EmbeddingJobType { ⋮---- type Error = anyhow::Error; fn try from value: &str - Result { ⋮---- "embedding generation" = Ok EmbeddingJobType::EmbeddingGeneration , "completion" = Ok EmbeddingJobType::Completion , ⋮---- pub struct EmbeddingArgs { ⋮---- impl EmbeddingArgs { pub fn with defaults self - Self { ⋮---- out uri: self.out uri.or Some self.uri.clone , out table: self.out table.or Some self.table.clone , ⋮---- pub struct ShowModelsArgs { ⋮---- pub struct MeasureModelSpeedArgs { Evidence: `lantern_cli/src/embeddings/cli.rs`
- **Mod** (source_file): pub mod cohere runtime; pub mod http runtime; pub mod openai runtime; pub mod ort runtime; pub mod runtime; pub mod utils; ⋮---- use cohere runtime::CohereRuntime; use openai runtime::OpenAiRuntime; use ort runtime::OrtRuntime; use runtime::EmbeddingRuntimeT; ⋮---- use super::cli::EmbeddingJobType; ⋮---- fn default logger text: &str { println! "{}", text ; ⋮---- pub enum Runtime { ⋮---- pub type LoggerFn = fn &str ; impl FromStr for Runtime { type Err = anyhow::Error; fn from str input: &str - Result { ⋮---- "ort" = Ok Runtime::Ort , "openai" = Ok Runtime::OpenAi , "cohere" = Ok Runtime::Cohere , ⋮---- type Error = anyhow::Error; fn try from value: &str - Result { ⋮---- impl ToString for Ru… Evidence: `lantern_cli/src/embeddings/core/mod.rs`
- **Mod** (source_file): use bytes::BytesMut; use core::get available runtimes; use futures::SinkExt; use rand::Rng; use std::sync::Arc; use std::time::Instant; ⋮---- use tokio::task::JoinHandle; use tokio util::sync::CancellationToken; ⋮---- use self::cli::EmbeddingJobType; use self::core::EmbeddingRuntime; ⋮---- pub mod cli; pub mod core; pub mod measure speed; ⋮---- struct EmbeddingRecord { ⋮---- impl EmbeddingRecord { fn from vec pk: String, value: Vec - EmbeddingRecord { ⋮---- if value.len 0 { ⋮---- buf.extend from slice "{".as bytes ; let row str: String = value.iter .map &x x.to string + "," .collect ; buf.extend from slice row str 0..row str.len - 1 .as bytes ; drop row str ; buf.extend from slice "}".as by… Evidence: `lantern_cli/src/embeddings/mod.rs`
- **Cli** (source_file): pub enum UMetricKind { ⋮---- impl UMetricKind { pub fn from ops ops: &str - Result { ⋮---- return Ok UMetricKind::L2sq ; ⋮---- return Ok UMetricKind::Cos ; ⋮---- return Ok UMetricKind::Hamming ; ⋮---- pub fn to ops &self - String { ⋮---- return "dist l2sq ops".to owned ; ⋮---- return "dist cos ops".to owned ; ⋮---- return "dist hamming ops".to owned ; ⋮---- pub fn from metric kind: &str - Result { ⋮---- pub fn from u32 metric kind: u32 - Result { ⋮---- pub fn to string &self - String { ⋮---- return "l2sq".to owned ; ⋮---- return "cos".to owned ; ⋮---- return "hamming".to owned ; ⋮---- pub fn value &self - MetricKind { ⋮---- pub fn sql function &self - String { ⋮---- return "l2sq dist".to ow… Evidence: `lantern_cli/src/external_index/cli.rs`
- **Mod** (source_file): pub mod cli; pub mod server; Evidence: `lantern_cli/src/external_index/mod.rs`
- **Cli** (source_file): use clap::Parser; ⋮---- pub struct HttpServerArgs { Evidence: `lantern_cli/src/http_server/cli.rs`
- **Mod** (source_file): use cli::HttpServerArgs; ⋮---- use tokio postgres::NoTls; use utoipa::OpenApi; use utoipa swagger ui::SwaggerUi; ⋮---- pub mod cli; mod collection; mod index; mod pq; mod search; mod setup; ⋮---- type PoolClient = deadpool::managed::Object ; ⋮---- struct AppPool { ⋮---- impl AppPool { fn new pool: Pool - AppPool { ⋮---- async fn get &self - Result { let client = self.inner.get .await; ⋮---- Err e = Err ErrorInternalServerError e , Ok client = Ok client , ⋮---- pub struct AuthCredentials { ⋮---- pub struct AppState { ⋮---- pub type AppData = web::Data ; ⋮---- async fn auth validator ⋮---- let data = req.app data:: .unwrap ; ⋮---- if creds.username != credentials.user id creds.password != cre… Evidence: `lantern_cli/src/http_server/mod.rs`
- **Cli** (source_file): use crate::external index::cli::UMetricKind; use clap::Parser; ⋮---- pub struct IndexAutotuneArgs { Evidence: `lantern_cli/src/index_autotune/cli.rs`
- **Mod** (source_file): use rand::Rng; ⋮---- pub mod cli; ⋮---- type GroundTruth = Vec , Vec ; ⋮---- struct IndexParams { ⋮---- struct AutotuneResult { ⋮---- fn create test table ⋮---- client.batch execute &format! ⋮---- let dims = client.query one &format! "SELECT ARRAY LENGTH v, 1 FROM {tmp table name} WHERE v IS NOT NULL LIMIT 1", , ⋮---- let dims: i32 = dims.get 0 ; ⋮---- let sample size = client.query one &format! "SELECT COUNT FROM {tmp table name}" , & ?; let sample size: i64 = sample size.get 0 ; ⋮---- Ok sample size as i32 ⋮---- fn export results ⋮---- let mut query = format! "INSERT INTO {result table full name} experiment id, ef, efc, m, recall, latency, build time VALUES " ; ⋮---- .iter .flat map row {… Evidence: `lantern_cli/src/index_autotune/mod.rs`
- **Mod** (source_file): pub enum LogLevel { ⋮---- pub struct Logger { ⋮---- impl Logger { pub fn new label: &str, level: LogLevel - Logger { ⋮---- label: label.to owned , ⋮---- pub fn print raw &self, msg: &str { println! "{}", msg ; ⋮---- pub fn info &self, msg: &str { ⋮---- println! " {} {}", &self.label, msg ; ⋮---- pub fn debug &self, msg: &str { ⋮---- println! " + {} {}", &self.label, msg ; ⋮---- pub fn warn &self, msg: &str { ⋮---- println! " ! {} {}", &self.label, msg ; ⋮---- pub fn error &self, msg: &str { eprintln! " X {} {}", &self.label, msg ; ⋮---- impl Clone for Logger { fn clone &self - Self { Logger::new &self.label, self.level.clone Evidence: `lantern_cli/src/logger/mod.rs`
- **Main** (source_file): mod cli; ⋮---- async fn main { use std::process; ⋮---- use clap::Parser; ⋮---- use tokio util::sync::CancellationToken; ⋮---- .install default .expect "Failed to install default CryptoProvider" ; ⋮---- main logger = Some logger.clone ; ⋮---- Some logger , ⋮---- let logger = main logger.as ref .unwrap ; logger.error &e.to string ; ⋮---- Ok ⋮---- embeddings::show available models &args, Some logger .await ⋮---- embeddings::show available runtimes Some logger ⋮---- embeddings::measure speed::start speed test &args, Some logger .await ⋮---- index autotune::autotune index &args, None, None, Some logger ⋮---- .unwrap ⋮---- tokio::task::spawn blocking move pq::quantize table args, None, None, Some… Evidence: `lantern_cli/src/main.rs`
- **Cli** (source_file): use clap::Parser; ⋮---- pub struct PQArgs { Evidence: `lantern_cli/src/pq/cli.rs`
- **Mod** (source_file): use crate::types::JOB CANCELLED MESSAGE; ⋮---- use codebook::CreateCodebookArgs; use quantization::QuantizeAndWriteVectorArgs; use rand::Rng; ⋮---- use std::time::Instant; ⋮---- pub mod cli; mod codebook; mod gcp batch; mod quantization; mod setup; ⋮---- type AnyhowVoidResult = Result ; pub type ProgressCbFn = Box ; ⋮---- fn report progress ⋮---- old progress.fetch add progress, Ordering::SeqCst ; let new progress = old progress.load Ordering::SeqCst ; logger.info &format! "Progress {new progress}%" ; if progress cb.is some { let cb = progress cb.as ref .unwrap ; cb new progress ; ⋮---- fn set and report progress ⋮---- let old progress value = old progress.load Ordering::SeqCst ; ⋮---- repo… Evidence: `lantern_cli/src/pq/mod.rs`
- **Types** (source_file): pub type AnyhowUsizeResult = Result ; pub type AnyhowVoidResult = Result ; pub type ProgressCbFn = Box ; Evidence: `lantern_cli/src/types.rs`
- **Mod** (source_file): pub mod test utils; ⋮---- pub fn quote ident str: &str - String { format! "\"{}\"", str.replace "\"", "\"\"" ⋮---- pub fn quote literal str: &str - String { let replaced = str.replace "'", "''" ; format! "'{}'", replaced ⋮---- pub fn get full table name schema: &str, table: &str - String { let schema = quote ident schema ; let table = quote ident table ; format! "{schema}.{table}" ⋮---- pub fn append params to uri db uri: &str, params: &str - String { let parts: Vec = db uri.split "/" .collect ; let mut db uri string = db uri.to string ; ⋮---- if let Some last part = parts.last { has params = last part.contains "?" ; ⋮---- db uri string.push str &format! "&{params}" ; ⋮---- db uri string.pu… Evidence: `lantern_cli/src/utils/mod.rs`
- **This requires the docker command be run in the lantern base director** (source_file): ARG VERSION=15.4 ARG PGVECTOR VERSION=0.5.0 ARG UBSAN= Evidence: `lantern_hnsw/scripts/sanitizers/Dockerfile`
- **Lantern** (source_file): CREATE FUNCTION hnsw handler internal RETURNS index am handler AS 'MODULE PATHNAME' LANGUAGE C; CREATE FUNCTION ldb generic dist real , real RETURNS real AS 'MODULE PATHNAME' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE; CREATE FUNCTION ldb generic dist integer , integer RETURNS real AS 'MODULE PATHNAME' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE; CREATE FUNCTION l2sq dist real , real RETURNS real AS 'MODULE PATHNAME' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE; CREATE FUNCTION cos dist real , real RETURNS real AS 'MODULE PATHNAME' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE; CREATE FUNCTION hamming dist integer , integer RETURNS integer AS 'MODULE PATHNAME' LANGUAGE C IMMUTABLE STRICT PARALLEL SAF… Evidence: `lantern_hnsw/sql/lantern.sql`
- **Hnsw** (source_file): static char hnswbuildphasename int64 phasenum ⋮---- static uint64 expected number of levels double num tuples in index, double mL ⋮---- static uint64 estimate number tuples accessed Oid index relation, double num tuples in index ⋮---- static uint64 estimate number blocks accessed uint64 num tuples in index, uint64 num pages, uint64 num tuples accessed ⋮---- static void hnswcostestimate PlannerInfo root, ⋮---- static bool hnswvalidate Oid opclassoid ⋮---- PGDLLEXPORT PG FUNCTION INFO V1 hnsw handler ; Datum hnsw handler PG FUNCTION ARGS attribute unused ⋮---- static float4 array dist ArrayType a, ArrayType b, usearch metric kind t metric kind ⋮---- static float8 vector dist Vector a, Vector… Evidence: `lantern_hnsw/src/hnsw.c`
- **Hnsw** (source_file): } HnswColumnType; ⋮---- PGDLLEXPORT void PG init void ; PGDLLEXPORT void PG fini void ; ⋮---- HnswColumnType GetColumnTypeFromOid Oid oid ; HnswColumnType GetIndexColumnType Relation index ; void DatumGetSizedArray Datum datum, HnswColumnType type, int dimensions, bool copy ; Evidence: `lantern_hnsw/src/hnsw.h`
- **Build** (source_file): static void AddTupleToUsearchIndex ItemPointer tid, ⋮---- static void BuildCallback ⋮---- static int GetArrayLengthFromExpression Expr expression, Relation heap, HeapTuple tuple ⋮---- static int GetArrayLengthFromHeap Relation heap, int indexCol, IndexInfo indexInfo ⋮---- int GetHnswIndexDimensions Relation index, IndexInfo indexInfo ⋮---- void CheckHnswIndexDimensions Relation index, Datum arrayDatum, int dimensions ⋮---- static int InferDimension Relation heap, IndexInfo indexInfo ⋮---- static void InitBuildState ldb HnswBuildState buildstate, Relation heap, Relation index, IndexInfo indexInfo ⋮---- static void FreeBuildState ldb HnswBuildState buildstate ⋮---- static void ScanTable ldb H… Evidence: `lantern_hnsw/src/hnsw/build.c`
- **Delete** (source_file): IndexBulkDeleteResult ldb ambulkdelete IndexVacuumInfo info, ⋮---- IndexBulkDeleteResult ldb amvacuumcleanup IndexVacuumInfo info, IndexBulkDeleteResult stats Evidence: `lantern_hnsw/src/hnsw/delete.c`
- **Insert** (source_file): static int16 hnsw generate new level size t connectivity ⋮---- bool ldb aminsert Relation index, Evidence: `lantern_hnsw/src/hnsw/insert.c`
- **Scan** (source_file): IndexScanDesc ldb ambeginscan Relation index, int nkeys, int norderbys ⋮---- void ldb amendscan IndexScanDesc scan ⋮---- void ldb amrescan IndexScanDesc scan, ScanKey keys, int nkeys, ScanKey orderbys, int norderbys ⋮---- bool ldb amgettuple IndexScanDesc scan, ScanDirection dir Evidence: `lantern_hnsw/src/hnsw/scan.c`
- **Usearch Storage** (source_file): uint32 t UsearchNodeBytes const metadata t metadata, int vector bytes, int level ; void usearch init node ⋮---- uint32 node tuple size char node, uint32 vector dim, const metadata t meta ; ⋮---- usearch label t label from node char node ; unsigned long level from node char node ; void reset node label char node ; ⋮---- ldb unaligned slot union t get node neighbors mut const metadata t meta, Evidence: `lantern_hnsw/src/hnsw/usearch_storage.hpp`
- **Daemon Autotune Test With Db** (source_file): use tokio util::sync::CancellationToken; ⋮---- async fn test daemon autotune with create index { ⋮---- setup test "test daemon autotune with create index" ⋮---- .unwrap ; ⋮---- .batch execute &format! ⋮---- let cancel token clone = cancel token.clone ; ⋮---- schema: " lantern extras internal".to owned , target db: Some vec! new connection uri , ⋮---- wait for completion ⋮---- &format! "SELECT COUNT =1 FROM pg indexes WHERE indexdef LIKE '%lantern hnsw%'" , ⋮---- cancel token.cancel ; Evidence: `lantern_cli/tests/daemon_autotune_test_with_db.rs`
- **Daemon Completion Test With Db** (source_file): use std::env; use tokio util::sync::CancellationToken; ⋮---- async fn test daemon completion init job { let api token = env::var "OPENAI TOKEN" .unwrap or "".to owned ; ⋮---- setup test "test daemon completion init job" .await.unwrap ; ⋮---- .batch execute &format! ⋮---- .unwrap ; ⋮---- let cancel token clone = cancel token.clone ; ⋮---- schema: " lantern extras internal".to owned , target db: Some vec! new connection uri , ⋮---- wait for completion ⋮---- &format! "SELECT COUNT =5 FROM {CLIENT TABLE NAME} WHERE num = substring title, 5 ::INT" , ⋮---- cancel token.cancel ; Evidence: `lantern_cli/tests/daemon_completion_test_with_db.rs`
- **Daemon Embeddings Test With Db** (source_file): use std::time::Duration; use tokio util::sync::CancellationToken; ⋮---- async fn test daemon embedding init job { ⋮---- setup test "test daemon embedding init job" .await.unwrap ; ⋮---- .batch execute &format! ⋮---- .unwrap ; ⋮---- let cancel token clone = cancel token.clone ; ⋮---- schema: " lantern extras internal".to owned , target db: Some vec! new connection uri , ⋮---- wait for completion ⋮---- &format! "SELECT COUNT =5 FROM {CLIENT TABLE NAME} WHERE title embedding IS NOT NULL" , ⋮---- cancel token.cancel ; ⋮---- async fn test daemon embedding job client insert listener { ⋮---- setup test "test daemon embedding job client insert listener" ⋮---- async fn test daemon embedding job clie… Evidence: `lantern_cli/tests/daemon_embeddings_test_with_db.rs`
- **Embedding Test With Db** (source_file): use tokio postgres::IsolationLevel; ⋮---- use tokio util::sync::CancellationToken; ⋮---- async fn setup db tables client: &mut Client, table name: &str { ⋮---- .build transaction .isolation level IsolationLevel::Serializable .read only false .start ⋮---- .unwrap ; ⋮---- .batch execute &format! ⋮---- .expect "Could not create necessarry tables" ; transaction.commit .await.unwrap ; ⋮---- async fn drop db tables client: &mut Client, table name: &str { ⋮---- .expect "Could not drop tables" ; ⋮---- async fn test embedding generation from db { let db url = env::var "DB URL" .expect " DB URL not specified" ; ⋮---- .expect "Can not connect to database" ; tokio::spawn async move { connection.await.u… Evidence: `lantern_cli/tests/embedding_test_with_db.rs`
- **Http Server Test With Db** (source_file): async fn drop db tables client: &mut Client { ⋮---- .batch execute &format! ⋮---- .expect "Could not drop tables" ; ⋮---- async fn test setup - Sender { let db uri = env::var "DB URL" .expect " DB URL not specified" ; start server db uri ⋮---- fn start server db uri: String - Sender { ⋮---- host: "127.0.0.1".to owned , ⋮---- username: Some "test".to owned , password: Some "test".to owned , ⋮---- .expect "Failed to start HTTP Server" ; ⋮---- match rx.try recv { ⋮---- async fn test cleanup { ⋮---- let mut db client, connection = tokio postgres::connect &db uri, NoTls .await.unwrap ; tokio::spawn async move { connection.await.unwrap } ; ⋮---- drop db tables &mut db client .await; ⋮---- async f… Evidence: `lantern_cli/tests/http_server_test_with_db.rs`
- **Pq Test With Db** (source_file): use lantern cli::pq; ⋮---- fn setup db tables client: &mut Client, table name: &str, range start: usize, range end: usize { ⋮---- .batch execute &format! ⋮---- .expect "Could not create necessarry tables" ; ⋮---- fn drop db tables client: &mut Client, table name: &str, codebook table name: &str { ⋮---- .expect "Could not drop tables" ; ⋮---- fn test full pq { let db url = env::var "DB URL" .expect " DB URL not specified" ; ⋮---- let codebook table name = get full table name " lantern internal", "pq pq test v" ; let mut db client = Client::connect &db url, NoTls .expect "Database connection failed" ; drop db tables &mut db client, &table name, &codebook table name ; setup db tables &mut db c… Evidence: `lantern_cli/tests/pq_test_with_db.rs`
- **Postgres symbols won't be available until runtime** (source_file): target.aarch64-apple-darwin rustflags = "-C", "link-args=-Wl,-rpath,/usr/local/lib/onnxruntime/lib" target.x86 64-apple-darwin rustflags = "-C", "link-args=-Wl,-rpath,/usr/local/lib/onnxruntime/lib" target.'cfg target os="macos" ' Postgres symbols won't be available until runtime rustflags = "-Clink-arg=-Wl,-undefined,dynamic lookup" env ORT STRATEGY = { value = "system" } ORT LIB LOCATION = { value = "/usr/local/lib/onnxruntime/lib" , relative = false } Evidence: `.cargo/config.toml`
- **BeforeElse : true** (source_file): Language: Cpp BasedOnStyle: Google Standard: Cpp11 Evidence: `.clang-format`
- **.gitignore** (source_file): !.github/ /dist/ /log/ /results/ /tmp check/ /sql/vector--?.?.?.sql regression. .o .so .bc .dll .dylib .obj .lib .exp .a test/tmp output/ .DS Store /build / data sanitizer .vscode/ .devcontainer/ .cache .cosine .env.rb .venv pycache lantern hnsw/build .idea/ /target .iml / .rs.bk Cargo.lock .ldb extras data .profraw Evidence: `.gitignore`
- **.gitmodules** (source_file): submodule "third party/usearch" path = lantern hnsw/third party/usearch url = https://github.com/ngalstyan4/usearch branch = pg-rebase Evidence: `.gitmodules`
- **Lantern Extras** (source_file): comment = 'lantern extras: Convenience functions for working with vector embeddings' default version = '@CARGO VERSION@' module pathname = '$libdir/lantern extras' relocatable = false superuser = true Evidence: `lantern_extras/lantern_extras.control`

## Rules the Host AI Must Follow

- **Treat this asset as pre-work context, not a runtime environment.**: The AI Context Pack contains only an evidence-backed understanding of the project, not the project's executable state. Evidence: `lantern_extras/README.md`, `lantern_hnsw/scripts/sanitizers/suppressions/README.md`, `CONTRIBUTING.md`
- **When answering the user, distinguish what can be previewed from what can only be verified after install.**: The consumer value of the pre-install experience comes from reducing bad installs and misjudgments, not from pretending to be a real run. Evidence: `lantern_extras/README.md`, `lantern_hnsw/scripts/sanitizers/suppressions/README.md`, `CONTRIBUTING.md`

## Questions the User Should Answer First

- Which host AI or local environment do you plan to use it in?
- Do you just want to experience the workflow first, or are you ready to actually install?
- What matters most to you: install cost, output quality, or conflicts with your existing rules?

## Acceptance Checks

- Every capability claim can be traced back to a file path in evidence_refs.
- AI_CONTEXT_PACK.md does not package previews as a real run.
- The user can understand who it fits, what it can do, how to start, and the risk boundaries within 3 minutes.

---

## Doramagic Context Augmentation

The following sections strengthen the repository context for a host AI. Human Manual data is a reading route, and pitfall notes become operating constraints.

## Human Manual Outline

Usage rule: this is only a reading route and salience signal, not factual authority. Concrete claims must still return to repo evidence or Claim Graph.

Host AI hard rules:
- Do not treat page titles, section order, summaries, or importance values as factual project evidence.
- When explaining the Human Manual outline, state that it is only a reading route or salience signal.
- Capability, installation, compatibility, runtime state, and risk claims must cite repo evidence, source paths, or Claim Graph.

- **Lantern Overview and Component Architecture**: importance `high`
  - source_paths: README.md, Cargo.toml, lantern_hnsw/CMakeLists.txt, lantern_hnsw/sql/lantern.sql, lantern_hnsw/src/hnsw.h
- **lantern_hnsw C Extension Internals**: importance `high`
  - source_paths: lantern_hnsw/src/hnsw.c, lantern_hnsw/src/hnsw.h, lantern_hnsw/src/hnsw/build.c, lantern_hnsw/src/hnsw/insert.c, lantern_hnsw/src/hnsw/delete.c
- **lantern_cli and lantern_extras Tooling**: importance `high`
  - source_paths: lantern_cli/src/main.rs, lantern_cli/src/cli.rs, lantern_cli/src/types.rs, lantern_cli/src/daemon/mod.rs, lantern_cli/src/daemon/cli.rs
- **Build, Installation, Release, and Known Failure Modes**: importance `high`
  - source_paths: README.md, CHANGELOG.md, CONTRIBUTING.md, .dockerignore, docker/Dockerfile

## Repo Inspection Evidence

- repo_clone_verified: true
- repo_inspection_verified: true
- repo_commit: `7df590ce19080df7dd8a8bbb0f9875fc324bb2ea`
- inspected_files: `README.md`

Host AI hard rules:
- Without repo_clone_verified=true, do not claim that the source code has been read.
- Without repo_inspection_verified=true, do not write README, docs, or package-file conclusions as facts.
- Without quick_start_verified=true, do not claim that the Quick Start path has run successfully.

## Doramagic Pitfall Constraints

These rules come from Doramagic discovery, validation, or compilation findings. The host AI must treat them as operating constraints, not background notes.

### Constraint 1: Capability evidence risk requires verification

- Trigger: README/documentation is current enough for a first validation pass.
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/lanterndata/lantern
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.

### Constraint 2: Security or permission risk requires verification

- Trigger: no_demo
- Host AI rule: Reproduce the official install and quickstart path in an isolated environment.
- Why it matters: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/lanterndata/lantern
- Hard boundary: Do not present this pitfall as solved, verified, or ignorable unless later evidence explicitly closes it.