Related Pages

Related topics: Five-Index Architecture, Core API Reference, Installation

Overview

YantrikDB is a cognitive memory database system designed to model, store, and reason about complex human mental states and behaviors. It provides a unified architecture for managing episodic memories, semantic knowledge, and procedural information while supporting advanced cognitive operations such as attention spreading, belief revision, and proactive suggestion surfacing.

The system bridges traditional database storage with cognitive science principles, enabling applications that require understanding of user intent, emotional states, preferences, and behavioral patterns.

Architecture Overview

YantrikDB follows a layered architecture that separates storage, cognitive processing, and query execution concerns.

graph TD
    A[Python API Layer] --> B[Query DSL Engine]
    B --> C[Cognition Module]
    C --> D[Working Set Cache]
    D --> E[SQLite Storage]
    C --> F[Conflict Resolution]
    C --> G[Proactive Surfacing]
    C --> H[Narrative Tracking]

Core Modules

Sources: crates/yantrikdb-core/src/cognition/state.rs:1-50

Cognitive Node Model

Every entity in YantrikDB is represented as a cognitive node with universal attributes that determine how it participates in reasoning, memory consolidation, and action selection.

Node Kinds

The system supports 15 distinct node types representing different mental constructs:

Sources: crates/yantrikdb-core/src/cognition/state.rs:150-180

Universal Cognitive Attributes

Every node carries 11 dimensions that govern its lifecycle and behavior:

graph LR
    A[Cognitive Node] --> B[confidence 0-1]
    A --> C[activation 0-1]
    A --> D[salience 0-1]
    A --> E[persistence 0-1]
    A --> F[valence -1 to 1]
    A --> G[urgency 0-1]
    A --> H[novelty 0-1]
    A --> I[volatility 0-1]
    A --> J[provenance]
    A --> K[evidence_count]

The default values for each node kind vary based on their expected characteristics:

Sources: crates/yantrikdb-core/src/cognition/state.rs:280-320

Cognitive Edge Model

Nodes connect through typed edges that encode semantic relationships and govern activation spreading.

Edge Kinds

There are 15 edge types in the cognitive graph:

Sources: crates/yantrikdb-core/src/cognition/state.rs:120-160

Edge Behavior Methods

Each edge kind provides metadata through dedicated methods:

• activation_transfer() - Returns the spreading activation factor (-1.0 to 1.0)
• is_inhibitory() - Boolean indicating suppression behavior
• is_epistemic() - Whether edge participates in belief revision
• is_causal() - Whether edge represents causal relationships

pub fn is_inhibitory(self) -> bool {
    self.activation_transfer() < 0.0
}

pub fn is_epistemic(self) -> bool {
    matches!(self, Self::Supports | Self::Contradicts)
}

pub fn is_causal(self) -> bool {
    matches!(self, Self::Causes | Self::Predicts | Self::Prevents)
}

Sources: crates/yantrikdb-core/src/cognition/state.rs:180-200

Cognitive Operators

The query DSL defines 10 operators that compose the cognitive processing pipeline:

Sources: crates/yantrikdb-core/src/cognition/query_dsl.rs:30-45

Operator Parameters

Each operator accepts typed parameters:

pub struct AttendOp {
    pub seeds: Vec<NodeId>,
    pub max_hops: u32,
    pub decay: f64,
}

pub struct RecallOp {
    pub top_k: usize,
    pub query: Option<String>,
    pub domain: Option<String>,
}

pub struct BelieveOp {
    pub evidence: EvidenceInput,
}

pub struct EvidenceInput {
    pub target: Option<NodeId>,
    pub observation: String,
    pub direction: i32,  // positive = confirming, negative = contradicting
}

Sources: crates/yantrikdb-core/src/cognition/query_dsl.rs:55-80

Execution Flow

graph TD
    A[Cognitive Query] --> B{Operator Type}
    B --> C[Attend]
    B --> D[Recall]
    B --> E[Believe]
    B --> F[Compare]
    B --> G[Plan]
    B --> H[Project]
    B --> I[Anticipate]
    B --> J[Assess]
    B --> K[CoherenceCheck]
    
    C --> L[Working Set Hydration]
    L --> M[Activation Boost on Seeds]
    M --> N[Spreading Activation]
    N --> O[StepOutput]

The executor processes operators by first hydrating the working set from SQLite, then executing operator-specific logic:

fn execute_attend(&self, op: &AttendOp) -> StepOutput {
    match self.db.hydrate_working_set(self.attention_config.clone()) {
        Ok(mut ws) => {
            let mut activated = 0;
            let mut top = Vec::new();
            for &seed in &op.seeds {
                if let Some(node) = ws.get_mut(seed) {
                    let new_activation = (node.attrs.activation + 0.3).min(1.0);
                    node.attrs.activation = new_activation;
                    top.push((seed, new_activation));
                    activated += 1;
                }
            }
            for &seed in &op.seeds {
                activated += ws.activate_and_spread(seed, 0.3);
            }
            StepOutput::Attend { nodes_activated: activated, top_activated: top }
        }
        Err(e) => StepOutput::Error { message: format!("Attend failed: {}", e) },
    }
}

Sources: crates/yantrikdb-core/src/engine/query_dsl.rs:100-130

Need Categories

The system models 8 categories of human needs:

pub fn from_str(s: &str) -> Self {
    match s {
        "informational" => Self::Informational,
        "social" => Self::Social,
        "emotional" => Self::Emotional,
        "organizational" => Self::Organizational,
        "creative" => Self::Creative,
        "health" => Self::Health,
        "financial" => Self::Financial,
        "professional" => Self::Professional,
        _ => Self::Informational,
    }
}

Sources: crates/yantrikdb-core/src/cognition/state.rs:10-45

Provenance and Reliability

Every cognitive node carries provenance metadata indicating its source:

pub fn reliability_prior(self) -> f64 {
    match self {
        Self::Told => 0.95,
        Self::Observed => 0.90,
        Self::Experimented => 0.85,
        Self::Consolidated => 0.80,
        Self::Extracted => 0.75,
        Self::Inferred => 0.60,
        Self::SystemDefault => 0.50,
    }
}

Sources: crates/yantrikdb-core/src/cognition/state.rs:220-250

Action Kinds and Costs

The system models 8 types of actions with associated base costs:

Higher cost indicates more disruption to the user.

Sources: crates/yantrikdb-core/src/cognition/state.rs:250-290

Conflict Resolution

The system detects and resolves conflicts between memories using policy-aware evaluation:

Conflict Types

Conflict Detection Flow

graph TD
    A[Memory Operations] --> B[Candidate Pair Generation]
    B --> C{Policy Check}
    C -->|overlap_allowed| D[Flag as Conflict]
    C -->|temporal_required| E{Time Validation}
    E -->|Valid| D
    E -->|Invalid| F[Apply Missing Time Severity]
    C -->|No Policy| G[Default Behavior]

The conflict resolution system queries namespace-specific policies:

SELECT overlap_allowed, temporal_required, missing_time_severity
FROM relation_policies
WHERE relation_type = ?1 AND (namespace = ?2 OR namespace = '*')
ORDER BY CASE WHEN namespace = ?2 THEN 0 ELSE 1 END

Sources: crates/yantrikdb-core/src/distributed/conflict.rs:50-80

Narrative Tracking

YantrikDB tracks narrative arcs to understand ongoing stories and life patterns:

Arc Types

Arc Lifecycle

graph LR
    A[Emerging] --> B[Active]
    B --> C[Paused]
    C -->|Resume| B
    C --> D[Resolved]
    C --> E[Abandoned]
    A -->|Quality| E

Chapter Types

Within arcs, chapters progress through phases:

Sources: crates/yantrikdb-core/src/cognition/narrative.rs:30-80

Personality Model

The system models personality across 8 dimensions affecting system behavior:

pub const DIMENSION_NAMES: [&'static str; 8] = [
    "curiosity", "proactivity", "caution", "warmth",
    "efficiency", "playfulness", "formality", "persistence",
];

pub fn similarity(&self, other: &Self) -> f64 {
    // Cosine similarity between personality vectors
    let mut dot = 0.0;
    let mut mag_a = 0.0;
    let mut mag_b = 0.0;
    for i in 0..Self::DIMENSIONS {
        let a = self.dimension(i);
        let b = other.dimension(i);
        dot += a * b;
        mag_a += a * a;
        mag_b += b * b;
    }
    // Normalized cosine similarity
    (dot / (mag_a.sqrt() * mag_b.sqrt())).clamp(-1.0, 1.0)
}

Sources: crates/yantrikdb-core/src/cognition/personality_bias.rs:50-90

User Receptivity

The system tracks user activity states to optimize notification timing:

Activity Levels

Notification Modes

Sources: crates/yantrikdb-core/src/cognition/receptivity.rs:20-70

Think() Configuration

The cognitive loop is configured via ThinkConfig:

pub struct ThinkConfig {
    pub importance_threshold: f64,
    pub decay_threshold: f64,
    pub max_triggers: usize,
}

Trigger Types

Sources: crates/yantrikdb-core/src/base/types.rs:100-150

Python API

The system exposes a Python interface for memory operations:

#[pyo3(signature = (
    query=None, embedding=None, top_k=10, memory_type=None, namespace=None,
    time_window=None, expand_entities=false, include_consolidated=false,
    skip_reinforce=false, domain=None, source=None
))]
fn query(
    &self,
    py: Python<'_>,
    query: Option<&str>,
    embedding: Option<Vec<f32>>,
    top_k: usize,
    // ... additional parameters
) -> PyResult<Vec<PyObject>>

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:50-90

Summary

YantrikDB provides a comprehensive cognitive memory architecture featuring:

• 15 cognitive node types with 11 universal attributes each
• 18 edge kinds with configurable activation transfer
• 10 cognitive operators for reasoning and context management
• 8 need categories for human motivation modeling
• 7 provenance levels with reliability priors
• 8 personality dimensions for behavioral adaptation
• 6 activity states for interruption optimization
• Policy-aware conflict resolution with namespace support
• Narrative arc tracking across 6 lifecycle states

The system balances persistent storage with working-set caching to support real-time cognitive operations while maintaining long-term memory integrity.

Related Pages

Related topics: Overview, Python Bindings, MCP Server Integration

Installation

This guide covers all supported methods for installing YantrikDB across different platforms and use cases.

Prerequisites

System Requirements

| Operating System | Linux, macOS, Windows | Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows ( Error with Openai API: output new_sensitive (1027)

Please check that you have set the OPENAI_API_KEY environment variable with a valid API key.

Related Pages

Related topics: Decoupled Write Path (LSM Architecture), Storage Engine, Core API Reference

Five-Index Architecture

Overview

The Five-Index Architecture is yantrikdb's multi-layered indexing system designed to support diverse query patterns across cognitive memory types. Each index layer specializes in a specific access pattern—vector similarity, temporal ordering, graph traversal, full-text search, and structured filtering—enabling the engine to retrieve relevant memories with minimal latency while maintaining consistency across the working set and persistent storage.

The architecture divides responsibility across five specialized index types:

Sources: crates/yantrikdb-core/src/engine/indices.rs:1-50

Architecture Diagram

graph TB
    subgraph "Query Interface"
        Q[RecallQuery]
    end
    
    subgraph "Five Index Layers"
        H[HNSW Index<br/>Vector ANN]
        D[Delta Index<br/>Working Set]
        G[Graph Index<br/>Relationships]
        S[Storage Index<br/>Persistent SQLite]
        R[Recall Index<br/>Federated Router]
    end
    
    subgraph "Data Sources"
        E[Embedding Cache]
        M[Memory Nodes]
        R2[Relational Tables]
    end
    
    Q --> R
    R --> H
    R --> D
    R --> G
    R --> S
    
    H --> E
    D --> M
    G --> M
    S --> R2

HNSW Index Layer

Purpose and Scope

The Hierarchical Navigable Small World (HNSW) index provides approximate nearest neighbor (ANN) search over high-dimensional embedding vectors. This layer powers semantic similarity queries, enabling the system to retrieve memories based on meaning rather than exact keyword matches.

Sources: crates/yantrikdb-core/src/vector/hnsw.rs:1-100

Key Components

The HNSW implementation in yantrikdb supports the following configuration parameters:

Query Flow

sequenceDiagram
    participant Q as Query
    participant H as HNSW Layer
    participant E as Embedding Cache
    participant R as Results
    
    Q->>H: RecallQuery with embedding
    H->>H: Layer 0 scan
    H->>H: Greedy search up layers
    H->>E: Fetch top_k candidates
    E-->>H: Candidate vectors
    H->>H: Re-rank by distance
    H-->>R: Ordered results

Delta Index Layer

Purpose and Scope

The Delta Index maintains a working set of recently inserted or updated records before they are flushed to persistent storage. This write buffer enables high-throughput ingestion while preserving query consistency for recent data.

Sources: crates/yantrikdb-core/src/vector/delta_index.rs:1-100

Write Path

When a new memory is created, the system:

1. Writes to the Delta Index immediately (low latency)
2. Appending to the HNSW structure if vector is present
3. Delaying SQLite flush until batch threshold

Consistency Model

The Delta Index implements a hybrid consistency model:

• Read-your-writes: Queries against recent data include Delta entries
• Staleness bound: Configurable flush interval (default: 1 second)
• Rollback support: Unflushed entries can be discarded on abort

graph LR
    A[Write Request] --> B{Delta Index<br/>In-Memory}
    B --> C{HNSW Update<br/>Immediate}
    C --> D[Query Path]
    B -.->|Flush| E[Storage Index<br/>SQLite]
    E --> D

Graph Index Layer

Purpose and Scope

The Graph Index manages typed relationships between memory nodes, supporting complex multi-hop queries. Each edge type has associated metadata including activation transfer factors and temporal validity windows.

Sources: crates/yantrikdb-core/src/knowledge/graph.rs:1-100 Sources: crates/yantrikdb-core/src/knowledge/graph_index.rs:1-100

Supported Edge Types

Graph Traversal API

// Core graph traversal via RecallQuery
RecallQuery::new(embedding)
    .top_k(10)
    .expand_entities(true)
    .max_hops(3)

Storage Index Layer

Purpose and Scope

The Storage Index provides durable persistence for all memory records using SQLite. This layer handles transaction management, crash recovery, and long-term storage optimization.

Sources: crates/yantrikdb-core/src/engine/storage.rs:1-100

Schema Overview

Query Parameters

Recall Index Layer

Purpose and Scope

The Recall Index acts as a federated query router that orchestrates multi-index searches. It combines results from HNSW, Delta, Graph, and Storage indices according to query parameters and relevance scoring.

Query Pipeline

flowchart TD
    A[RecallQuery] --> B[Parse Parameters]
    B --> C{HNSW Index}
    B --> D{Delta Index}
    B --> E{Graph Index}
    B --> F{Storage Index}
    
    C --> G[Result Merge]
    D --> G
    E --> G
    F --> G
    
    G --> H[Re-rank by Score]
    H --> I[Top-K Selection]
    I --> J[Return Ordered Results]

Query Construction

let q = RecallQuery::new(embedding)
    .top_k(10)
    .memory_type("episodic")
    .namespace("work")
    .time_window(start_ts, end_ts)
    .expand_entities(true)
    .include_consolidated(false);

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:50-80

Index Synchronization

Write Ordering

All index updates follow a strict ordering guarantee:

1. Delta Index receives write first (primary)
2. HNSW Index updated for vector-bearing records
3. Graph Index updated for edge-creating operations
4. Storage Index flush queued for background persistence

Failure Recovery

Performance Characteristics

Configuration

The Five-Index system is configured via ThinkConfig:

Summary

The Five-Index Architecture enables yantrikdb to handle diverse cognitive memory workloads by specializing each index for its access pattern. The HNSW layer provides fast semantic search, Delta absorbs write bursts, Graph manages relationships, Storage ensures durability, and Recall federates queries across all layers. This design allows the system to balance latency, throughput, and consistency according to workload characteristics.

Related Pages

Related topics: Five-Index Architecture, Storage Engine

Decoupled Write Path (LSM Architecture)

Overview

The Decoupled Write Path is the core write infrastructure of yantrikdb, implementing a Log-Structured Merge-tree (LSM) architecture that separates write operations from indexing and compaction. This design ensures high write throughput while maintaining read performance through asynchronous background compaction.

The architecture is built on the principle that write operations must be as fast as possible, deferring expensive vector indexing work to background processes. This decoupled approach prevents write operations from blocking on indexing operations, enabling the system to handle high concurrency workloads without regression.

Sources: CONCURRENCY.md

Architecture Components

DeltaIndex: The Write Buffer

The DeltaIndex is the primary write buffer in the LSM architecture. It provides lock-free append operations for new entries and tombstone operations for deletions.

graph TD
    subgraph WritePath["Write Path"]
        W[Write Request] --> DI[DeltaIndex]
        DI --> |append| DE[DeltaEntry]
        DE --> |O1 push| PendingVec[Pending Vec]
    end
    
    subgraph CompactionPath["Background Compaction"]
        PendingVec --> |seal| CT[Cold Tier]
        CT --> |clone-rebuild| NH[New HnswIndex]
        NH --> |install| CurrentCold[Current Cold Tier]
    end
    
    subgraph ReadPath["Read Path"]
        CurrentCold --> |ArcSwap| RR[Read Replicas]
    end

DeltaIndex Write Operations:

Sources: crates/yantrikdb-core/src/vector/delta_index.rs

Two-Tier Storage Model

The storage model consists of two tiers:

1. Hot Tier (DeltaIndex): Contains all recent writes and tombstones not yet compacted
2. Cold Tier (HnswIndex): Immutable, compacted index containing historical data

graph LR
    subgraph HotTier["Hot Tier - DeltaIndex"]
        D1[DeltaEntry 1]
        D2[DeltaEntry 2]
        D3[DeltaEntry N]
    end
    
    subgraph ColdTier["Cold Tier - HnswIndex"]
        H1[HnswIndex<br/>immutable]
        H2[HnswIndex<br/>immutable]
    end
    
    HotTier --> |periodic| ColdTier

Invariant: The cold tier MUST use ArcSwap<HnswIndex> for lock-free reader access. Replacing with RwLock<HnswIndex> or Mutex<HnswIndex> causes read latency regression.

Sources: CONCURRENCY.md

Concurrency Rules

The write path enforces strict concurrency rules to prevent deadlocks and ensure forward progress under high write load.

Rule 1: Foreground Writes Must Be O(1)

All foreground write operations MUST only touch O(1) data structures:

Sources: CONCURRENCY.md

Rule 2: Background Compaction Isolation

Background compaction MUST NOT share lock primitives with foreground writes:

sequenceDiagram
    participant FW as Foreground Write
    participant DI as DeltaIndex
    participant CP as Compactor
    participant HI as HnswIndex
    
    FW->>DI: append(entry)
    Note over DI: Brief RwLock write<br/>O(1) push
    
    CP->>DI: seal_delta_for_compaction()
    Note over DI: Brief lock for seal
    
    CP->>CP: clone cold + sealed entries
    Note over CP: No locks held here
    
    CP->>HI: ArcSwap new index
    Note over HI: Brief lock for install

Compactor Responsibilities:

1. Call seal_delta_for_compaction() to get a stable snapshot
2. Perform HNSW rebuild off the hot path
3. Install new cold tier via ArcSwap

Sources: CONCURRENCY.md

Rule 3: Visible Sequence Tracking

The visible_seq map tracks the minimum sequence number visible to readers per namespace, enabling read-your-writes (RYW) semantics.

// Type: DashMap<String, AtomicU64>
visible_seq: DashMap<String, AtomicU64>

// Fast path reads (lock-free)
get(ns).map(|e| e.load(Acquire))

// Fast path writes
get(ns).fetch_max(seq, Release)

Sources: CONCURRENCY.md

Write Operations

Standard Write Flow

graph TD
    Start[Write Request] --> Validate{Validate}
    Validate --> |valid| Seq[assign_seq]
    Validate --> |invalid| Reject[Reject]
    
    Seq --> SQL[SQL SAVEPOINT]
    SQL --> Delta[DeltaIndex::append]
    Delta --> Bump[bump_visible_seq]
    Bump --> Commit[Commit Transaction]
    Commit --> Done[Return to Client]
    
    Reject --> Fail[Return Error]

Record With RID Pattern

All write operations follow the record_with_rid pattern:

// Pattern for all write primitives
fn write_operation(&self, ...) -> Result<RecordId> {
    // 1. SQL with SAVEPOINT for rollback
    let rid = sql_transaction(|| {
        // 2. Append to DeltaIndex (O(1) push)
        self.delta_index.append(entry)?;
        Ok(assigned_rid)
    })?;
    
    // 3. Bump visible sequence
    self.bump_visible_seq(namespace, seq)?;
    
    Ok(rid)
}

Sources: CONCURRENCY.md

Sequence Number Assignment

The assign_seq function uses atomic operations for lock-free sequence generation:

// Atomic fetch_add or fetch_max
let seq = self
    .seq_counter
    .fetch_add(1, std::sync::atomic::Ordering::Relaxed);

This ensures each write receives a unique, monotonically increasing sequence number without contention.

Compaction Process

Compaction Lifecycle

graph LR
    subgraph Phase1["Phase 1: Seal"]
        A[Active Delta] --> B[Seal Delta]
        B --> C[Frozen Snapshot]
    end
    
    subgraph Phase2["Phase 2: Rebuild"]
        C --> D[Clone Cold Hnsw]
        D --> E[Merge Sealed Entries]
        E --> F[Build New Hnsw]
    end
    
    subgraph Phase3["Phase 3: Install"]
        F --> G[ArcSwap Install]
        G --> H[New Current Cold]
    end

Compaction Rules

Sources: CONCURRENCY.md

Compaction Triggers

The system triggers compaction based on configurable policies:

Sources: crates/yantrikdb-core/src/base/types.rs

Read-Your-Writes Semantics

Recall With Sequence

The recall_with_seq method enables clients to wait for their writes to become visible:

pub fn recall_with_seq(
    &self,
    query_embedding: &[f32],
    top_k: usize,
    min_seq: u64,           // Sequence from write operation
    namespace: Option<&str>,
    timeout: Duration,
) -> Result<Vec<RecallResult>> {
    let ns = namespace.unwrap_or("default");
    
    // Wait for visible_seq to reach min_seq
    self.wait_for_visible_seq(ns, min_seq, timeout)?;
    
    // Safe to recall - all writes up to min_seq are visible
    self.recall(query_embedding, top_k, ...)
}

Sources: crates/yantrikdb-core/src/engine/recall.rs

Visible Sequence Wait

sequenceDiagram
    participant C as Client
    participant VS as VisibleSeq Map
    participant REC as Recall Engine
    
    C->>VS: load current seq for namespace
    Note over VS: AtomicU64 load
    VS-->>C: current_seq
    
    alt current_seq < min_seq
        C->>C: wait_for_visible_seq()
        loop until visible or timeout
            C->>VS: load current seq
            VS-->>C: current_seq
        end
    end
    
    C->>REC: recall(...)
    REC-->>C: Results (guaranteed visible)

Materializer Integration

The materializer component coordinates between the write path and the cognitive layer, processing update operations extracted from natural language input.

graph TD
    subgraph Input["Input Processing"]
        NL[Natural Language] --> EX[Extractor]
        EX --> OT[Operation Templates]
    end
    
    subgraph Write["Write Path"]
        OT --> UW[UpdateOps]
        UW --> DI[DeltaIndex]
        DI --> SEQ[Sequence Assignment]
        SEQ --> VS[VisibleSeq Update]
    end
    
    subgraph Cognitive["Cognitive Layer"]
        VS --> MAT[Materializer]
        MAT --> ST[State Update]
        ST --> GP[Graph Propagation]
    end

Sources: crates/yantrikdb-core/src/engine/materializer.rs Sources: crates/yantrikdb-core/src/cognition/extractor.rs

Configuration

ThinkConfig Parameters

The cognition loop configuration affects compaction behavior:

Sources: crates/yantrikdb-core/src/base/types.rs

Memory Query Options

The Python bindings expose configuration for recall operations:

#[pyo3(signature = (
    query=None, embedding=None, top_k=10, memory_type=None, namespace=None,
    time_window=None, expand_entities=false, include_consolidated=false,
    skip_reinforce=false, domain=None, source=None
))]

Sources: crates/yantrikdb-python/src/py_engine/memory.rs

Performance Characteristics

Write Path Guarantees

Compaction Guarantees

Read Path Guarantees

Error Handling

Conflict Resolution

The system tracks conflicts between memories for resolution:

pub struct Conflict {
    pub conflict_id: String,
    pub conflict_type: String,        // identity_fact, preference, temporal
    pub priority: String,              // critical, high, medium, low
    pub memory_a: String,
    pub memory_b: String,
    pub entity: Option<String>,
    pub detected_at: f64,
    pub resolution_note: Option<String>,
}

Sources: crates/yantrikdb-core/src/base/types.rs

Related Documentation

• Concurrency Rules - Detailed concurrency invariants
• Decoupled Write Path RFC - Design rationale
• DeltaIndex Implementation - Source code
• Materializer - Write coordination

Related Pages

Related topics: Five-Index Architecture, Decoupled Write Path (LSM Architecture)

Storage Engine

Overview

The YantrikDB Storage Engine is the core persistence layer responsible for storing, retrieving, and managing memory data in the SQLite database. It handles encrypted text storage, metadata management, storage tier organization, and integrates with the materialization subsystem for asynchronous processing of memory operations.

The storage engine operates as part of the broader engine module and maintains close integration with:

• The Record System for writing memories
• The Recall System for querying memories
• The Materialization Pipeline for asynchronous operation processing
• The Encryption Layer for secure text storage

Core Data Models

Memory Structure

The central data structure managed by the storage engine is the Memory struct, which encapsulates all attributes of a stored memory:

pub struct Memory {
    pub rid: String,                      // Unique record identifier
    pub memory_type: String,             // episodic, semantic, procedural, etc.
    pub text: String,                    // Decrypted memory content
    pub created_at: f64,                 // Creation timestamp
    pub importance: f64,                 // Importance score [0.0, 1.0]
    pub valence: f64,                    // Emotional valence [-1.0, 1.0]
    pub half_life: f64,                  // Decay half-life in seconds
    pub last_access: f64,                // Last access timestamp
    pub access_count: u32,               // Number of times accessed
    pub consolidation_status: String,     // Current consolidation state
    pub storage_tier: String,             // hot, warm, cold, frozen
    pub consolidated_into: Option<String>,// RID of consolidated memory
    pub metadata: serde_json::Value,     // Encrypted JSON metadata
    pub namespace: String,               // Logical namespace partition
    pub certainty: f64,                  // Belief certainty [0.0, 1.0]
    pub domain: String,                   // Domain classification
    pub source: String,                   // Provenance source type
    pub emotional_state: Option<String>, // Associated emotional context
    pub session_id: Option<String>,      // Session identifier
    pub due_at: Option<f64>,             // Due timestamp for tasks
    pub temporal_kind: Option<String>,   // Temporal classification
}

Sources: engine/lifecycle.rs:200-225

Storage Tiers

YantrikDB implements a tiered storage architecture to optimize memory access patterns:

Sources: engine/lifecycle.rs:214

Consolidation Status

Memories maintain a consolidation status indicating their state in the memory consolidation lifecycle:

Each status carries a reliability prior that affects belief revision:

pub fn reliability_prior(self) -> f64 {
    match self {
        Self::Told => 0.95,          // User explicitly stated
        Self::Observed => 0.90,      // Directly observed
        Self::Experimented => 0.85,  // Controlled experiment
        Self::Extracted => 0.75,     // External documents
        Self::Inferred => 0.60,      // Pattern inference
        Self::Consolidated => 0.80,  // Multi-source merge
        Self::SystemDefault => 0.50, // Defaults
    }
}

Sources: cognition/state.rs:180-192

Storage Architecture

High-Level Architecture

graph TD
    subgraph "Python API Layer"
        PYM[py_engine/memory.rs]
    end
    
    subgraph "Engine Core"
        REC[Record System]
        RCL[Recall System]
        MAT[Materialization Pipeline]
    end
    
    subgraph "Storage Layer"
        ENG[Engine Instance]
        SQL[(SQLite Database)]
        CRE[Encryption Layer]
    end
    
    PYM --> REC
    PYM --> RCL
    REC --> ENG
    REC --> MAT
    MAT --> ENG
    ENG --> SQL
    ENG --> CRE
    CRE --> SQL

Encryption Integration

Text fields are encrypted before storage and decrypted on retrieval to ensure data privacy:

let text = self.decrypt_text(&row.2)?;
let meta_str = self.decrypt_text(&row.12)?;
let metadata: serde_json::Value = serde_json::from_str(&meta_str)
    .unwrap_or(serde_json::Value::Object(Default::default()));

Sources: engine/lifecycle.rs:210-214

Record Operations

Recording a Memory

The storage engine provides the record() method for storing new memories with automatic embedding:

db.record(
    text,           // Memory content
    memory_type,    // episodic, semantic, etc.
    importance,     // Importance score
    valence,        // Emotional valence
    half_life,      // Decay half-life
    &meta,          // JSON metadata
    &emb,           // Embedding vector
    namespace,      // Logical partition
    certainty,      // Belief certainty
    domain,         // Domain classification
    source,         // Provenance source
    emotional_state, // Emotional context
)

Sources: py_engine/memory.rs:45-60

Python Bindings

The Python API exposes record functionality through py_engine/memory.rs:

# Record a memory with auto-embedding
db.record(
    text="Meeting with John at 3pm",
    memory_type="episodic",
    namespace="work",
    importance=0.8,
    valence=0.5,
)

The record() method accepts these parameters:

Sources: py_engine/memory.rs:30-60

Recall Operations

Querying Memories

The recall system retrieves memories based on embedding similarity and filters:

db.recall(
    &emb,                    // Query embedding
    top_k,                   // Number of results
    time_window,             // Optional time filter
    memory_type,             // Type filter
    include_consolidated,    // Include consolidated memories
    expand_entities,         // Expand entity references
    query,                   // Optional text query
    skip_reinforce,          // Skip reinforcement learning
    namespace,               // Namespace filter
    domain,                  // Domain filter
    source,                  // Source filter
)

Sources: py_engine/memory.rs:105-120

Recall Query Builder

The RecallQuery struct provides a fluent interface for building recall queries:

let mut q = yantrikdb_core::RecallQuery::new(emb).top_k(top_k);
if let Some(mt) = memory_type {
    q = q.memory_type(mt);
}
if let Some(ns) = namespace {
    q = q.namespace(ns);
}
if let Some(tw) = time_window {
    q = q.time_window(tw.0, tw.1);
}

Sources: py_engine/memory.rs:145-155

Recall Parameters

Sources: py_engine/memory.rs:65-80

Materialization Pipeline

Overview

The materialization pipeline handles asynchronous processing of memory operations to ensure durability and consistency. It operates in phases:

graph LR
    A[Write Ops] --> B[Phase 3: Record/Rollback]
    B --> C[Phase 4.1: Materialize Pending]
    C --> D[Phase 4.2: Apply Updates]
    D --> E[Phase 4.3: Post-Record Materialization]
    E --> F[Applied]

Phase 3: Record Materialization

Handles synchronous recording of operations with rollback capability:

"record" | "forget" | "relate" | "correct" | "consolidate" => {
    tracing::trace!(
        target: "yantrikdb::ingest::materialize",
        op_id = %op_id,
        op_type = %op_type,
        "phase 3 stub: marking pending op as applied without inline materialization"
    );
    if self.mark_op_applied(op_id)? {
        applied += 1;
    }
}

Sources: engine/stats.rs:95-105

Phase 4.1-4.2: Update Operations

For update operations like task completion and status changes:

"create_task" | "update_task_status" | "create_goal" | "update_goal" 
| "record_belief" | "relate_belief" | "update_preference" 
| "record_need" | "record_emotion" => {
    // Attempt materialization
    match materialize_fn(payload) {
        Ok(()) => {
            if self.mark_op_applied(op_id)? {
                applied += 1;
            }
        }
        Err(e) => {
            tracing::warn!(
                target: "yantrikdb::ingest::materialize",
                op_id = %op_id,
                error = %e,
                "post-record-with-rid materialization failed; leaving pending for retry"
            );
        }
    }
}

Sources: engine/stats.rs:70-90

Phase 4.3: Post-Record Materialization

Handles entity and relation extraction that runs on the materializer thread to avoid blocking the foreground caller:

Mirrors the post-INSERT entity/relation extraction loop that used to live on the foreground record() path. Now runs on the materializer thread so the foreground caller is not blocked on the unbounded loop count.

Sources: engine/stats.rs:130-135

Conflict Detection

The storage engine integrates with the conflict detection system for distributed scenarios:

// Phase 2: Evaluate each candidate pair with policy awareness
for (src, rel_type, dst1, dst2, vf1, vt1, vf2, vt2, namespace) in &candidates {
    if conflicts.len() >= max_conflicts {
        break;
    }

    // RFC 006 Phase 3: check relation policy before flagging
    let policy: Option<(bool, bool, String)> = {
        let conn = db.conn();
        conn.query_row(
            "SELECT overlap_allowed, temporal_required, missing_time_severity \
             FROM relation_policies \
             WHERE relation_type = ?1 AND (namespace = ?2 OR namespace = '*') \
             ORDER BY CASE WHEN namespace = ?2 THEN 0 ELSE 1 END \
             LIMIT 1",
            params![rel_type, namespace],
            |row| { ... }
        )
    };
}

Sources: distributed/conflict.rs:85-105

Conflict Types

Sources: base/types.rs:50-60

Retrieval by ID

The get_memory_by_rid() method retrieves a specific memory by its record ID:

pub fn get_memory_by_rid(&self, rid: &str) -> Result<Option<Memory>> {
    let result = conn.query_row(
        "SELECT rid, memory_type, text, created_at, importance, valence,
                half_life, last_access, access_count, consolidation_status,
                storage_tier, consolidated_into, metadata, namespace,
                certainty, domain, source, emotional_state, session_id,
                due_at, temporal_kind
         FROM memories WHERE rid = ?1",
        params![rid],
        |row| Ok((...))  // 21 columns mapped
    )?;
    
    // Decrypt and deserialize
    let text = self.decrypt_text(&row.2)?;
    let meta_str = self.decrypt_text(&row.12)?;
    let metadata: serde_json::Value = serde_json::from_str(&meta_str)?;
    
    Ok(Some(Memory { ... }))
}

Sources: engine/lifecycle.rs:195-225

Query Interface

The storage engine provides a flexible query interface combining text and embedding search:

# Query memories with combined text and embedding search
results = db.query(
    query="team meeting",
    embedding=None,  # Auto-generate from query
    top_k=10,
    memory_type="episodic",
    namespace="work",
    time_window=(start_ts, end_ts),
    expand_entities=True,
    include_consolidated=False,
)

Sources: py_engine/memory.rs:85-100

Query vs Recall

Error Handling

The storage engine uses Rust's Result type for error handling with the following patterns:

.ok_or_else(|| PyRuntimeError::new_err("YantrikDB is closed"))

Errors are propagated through the Python bindings using the map_err function which converts Rust errors to Python exceptions.

Sources: py_engine/memory.rs:40

Performance Considerations

Memory Retrieval Optimization

1. Encryption on-demand: Text fields are only decrypted when accessed
2. Lazy metadata parsing: JSON metadata is parsed only when needed
3. Storage tiering: Frequently accessed memories can be promoted to hot tier
4. Consolidation filtering: include_consolidated=false skips consolidation lookups

Asynchronous Materialization

Post-record operations run on a background materializer thread to prevent foreground blocking:

Now runs on the materializer thread so the foreground caller is not blocked on the unbounded loop count (5-15...)

Sources: engine/stats.rs:135-138

Related Systems

API Reference Summary

Core Methods

Data Structures

Related Pages

Related topics: Overview, Cognition Layer, Conflict Detection and Resolution

Core API Reference

This page documents the core engine API of yantrikdb, covering the fundamental operations for memory storage, retrieval, graph relationships, and cognitive processing.

Overview

The Core API provides the foundational primitives for building personal memory systems. It consists of four primary subsystems:

Sources: crates/yantrikdb-core/src/engine/mod.rs

Architecture Overview

graph TD
    A[User Input] --> B[Record API]
    A --> C[Recall API]
    A --> D[Graph API]
    B --> E[(SQLite Storage)]
    C --> E
    D --> E
    E --> F[Cognitive Engine]
    F --> G[Think Loop]
    G --> H[Consolidation]
    G --> I[Conflict Detection]
    G --> J[Pattern Mining]

Memory Data Model

Core Memory Structure

The fundamental unit of storage in yantrikdb is the Memory struct, defined in lifecycle.rs:

| valence | f64 | Emotional valence [- Error with Openai API: output new_sensitive (1027)

Please check that you have set the OPENAI_API_KEY environment variable with a valid API key.

Related Pages

Related topics: Core API Reference, Conflict Detection and Resolution

Cognition Layer

The Cognition Layer is the reasoning and knowledge management subsystem of yantrikdb. It provides cognitive operations for belief revision, goal planning, intent recognition, pattern detection, and proactive user assistance. The layer orchestrates a pipeline of cognitive operators that process user interactions, external observations, and system events to maintain a dynamic model of user needs, goals, and preferences.

Architecture Overview

The Cognition Layer operates as a staged pipeline that transforms raw observations into structured cognitive entities (beliefs, goals, tasks, routines, intents) and surfaces actionable insights to the user at appropriate moments.

graph TD
    subgraph Input
        Obs[User Observation] --> Extr[Extractor]
        Ev[Evidence Input] --> Extr
    end
    
    subgraph "Cognitive Pipeline"
        Extr --> Ops[Operator Pipeline]
        Ops --> Attend[Attend]
        Ops --> Recall[Recall]
        Ops --> Believe[Believe]
        Ops --> Compare[Compare]
        Ops --> Plan[Plan]
        Ops --> Project[Project]
        Ops --> Anticipate[Anticipate]
        Ops --> Assess[Assess]
        Ops --> Coherence[Coherence Check]
    end
    
    subgraph "Working Memory"
        Attend --> WS[Working Set]
        Recall --> WS
    end
    
    subgraph "Long-term Store"
        WS <--> KG[Knowledge Graph]
        KG --> Beliefs[Beliefs]
        KG --> Goals[Goals]
        KG --> Routines[Routines]
    end
    
    subgraph "Output"
        Coherence --> Surf[Surfacing]
        Surf --> Suggest[Proactive Suggestion]
        Surf --> SurfaceMode[Surface Modes]
    end

Node Types

The Cognition Layer manages a graph of cognitive nodes, each representing a distinct aspect of user state and knowledge.

NodeKind Classification

Sources: state.rs:350-370

Cognitive Attributes

Every cognitive node carries a universal attribute set defining its dynamic state:

Sources: state.rs:280-330

Provenance System

The provenance system tracks the source and reliability of cognitive nodes, enabling appropriate trust calibration during reasoning.

Provenance Types

Sources: state.rs:220-240

Edge Types and Activation Spreading

Relationships between cognitive nodes are represented as typed edges with associated activation transfer coefficients that govern spreading activation dynamics.

Sources: state.rs:150-180

Cognitive Operators

The reasoning pipeline executes a sequence of cognitive operators in priority order. Each operator performs a specific reasoning function.

Operator Priorities

Sources: query_dsl.rs:40-55

Attend Operator

The Attend operator focuses attention on seed nodes and propagates activation through the knowledge graph.

pub struct AttendOp {
    pub seeds: Vec<NodeId>,      // Starting nodes for activation
    pub max_hops: u32,           // Maximum propagation depth
    pub decay: f64,              // Activation decay per hop
}

Sources: query_dsl.rs:65-70

Execution behavior:

• Seeds receive a +0.3 activation boost (capped at 1.0)
• Activation spreads through edges with configurable decay
• Returns count of activated nodes and top-activated node list

Sources: engine/query_dsl.rs:180-210

Recall Operator

The Recall operator retrieves relevant memories from long-term storage into the working set.

pub struct RecallOp {
    pub top_k: usize,           // Maximum results
    pub query: Option<String>,  // Text query
    pub domain: Option<String>, // Filter by domain
}

Believe Operator

The Believe operator integrates new evidence into the belief system using Bayesian revision.

pub struct BelieveOp {
    pub evidence: EvidenceInput,  // New observation to integrate
}

pub struct EvidenceInput {
    pub target: Option<NodeId>,   // Target belief or create new
    pub observation: String,       // The evidence
    pub direction: f64,           // +1 = confirming, -1 = contradicting
}

Compare Operator

Compares candidate actions or beliefs against constraints and preferences to select optimal choices.

Plan Operator

Executes means-ends reasoning to generate action sequences that advance specified goals.

Project Operator

Performs forward simulation to predict outcomes of potential action sequences.

Anticipate Operator

Identifies opportunities and risks based on current state and detected patterns.

Assess Operator

Evaluates overall system health, belief consistency, and goal progress.

Coherence Check

Validates logical consistency across beliefs and detects conflicting information.

User State Modeling

Activity Type

The system tracks the user's current activity level to calibrate interruption costs and suggestion timing.

Sources: receptivity.rs:25-50

Need Categories

User needs are classified according to a needs-based taxonomy:

Sources: state.rs:10-25

Action Types

Cognitive agents can perform actions of different kinds, each with associated base costs:

Sources: state.rs:100-130

Task Lifecycle

Tasks move through a defined status workflow:

graph LR
    P[Pending] --> IP[InProgress]
    IP --> C[Completed]
    IP --> B[Blocked]
    P --> CAN[Cancelled]
    B --> IP
    CAN --> P

Task Status

Sources: state.rs:200-230

Surfacing System

The surfacing system determines when and how to present proactive suggestions to the user.

Surface Modes

Suppression Reasons

Suggestions may be suppressed for various reasons:

Sources: surfacing.rs:15-30

ProactiveSuggestion Structure

pub struct ProactiveSuggestion {
    pub agenda_id: AgendaId,       // Source agenda item
    pub description: String,       // Human-readable text
    pub kind: AgendaKind,          // Type of open loop
    pub mode: SurfaceMode,         // How prominently to show
    pub reason: SurfaceReason,     // Why being surfaced
    pub confidence: f64,           // Relevance score [0,1]
    pub urgency: f64,              // Time sensitivity [0,1]
}

Conflict Detection

The system detects and manages conflicts between memories and beliefs.

Conflict Types

Sources: types.rs:180-210

Conflict Resolution

Conflicts are resolved through a policy-aware process that checks namespace-specific policies before flagging inconsistencies. Resolution strategies include:

• Timestamp-based: Newer observation wins
• Source-based: Higher provenance reliability wins
• Evidence count: More supporting observations wins
• Manual resolution: User intervention required for critical conflicts

Natural Language Understanding

The extractor component converts free-text observations into structured cognitive operations:

Sources: extractor.rs:150-180

Narrative Arc Tracking

The system maintains narrative structures to track ongoing storylines in the user's life:

Arc Status

Chapter Types

Narrative arcs are structured into chapters:

Sources: narrative.rs:50-80

Priority Levels

Cognitive entities use a priority tier system for urgency-based processing:

Sources: state.rs:240-260

Related Pages

Related topics: Cognition Layer, Core API Reference

Conflict Detection and Resolution

The Conflict Detection and Resolution system is a core cognitive subsystem within yantrikdb that identifies, categorizes, and resolves contradictions between beliefs, memories, and other cognitive nodes stored in the knowledge graph. This system ensures the internal consistency of the user's cognitive model by detecting conflicts, prioritizing them based on severity, and applying appropriate resolution strategies.

Overview

The conflict resolution system operates as part of the broader cognitive engine, integrated with the coherence checking pipeline. When contradictions are detected between nodes in the belief network, the system evaluates the evidence supporting each conflicting belief and automatically or semi-automatically resolves the conflict.

Sources: crates/yantrikdb-core/src/cognition/coherence.rs

Conflict Types

Conflicts in yantrikdb are categorized into five distinct types, each with different priority levels and default handling strategies.

Sources: crates/yantrikdb-core/src/base/types.rs

pub enum ConflictType {
    IdentityFact,
    Preference,
    Temporal,
    Consolidation,
    Minor,
}

Each conflict type has an associated default priority that determines how urgently it should be addressed in the surfacing queue.

Conflict Data Model

The Conflict struct captures all metadata about a detected conflict.

Conflict Structure

Sources: crates/yantrikdb-core/src/base/types.rs

Conflict Resolution Strategies

When a conflict is resolved, the system generates a ConflictResolutionResult containing details of the resolution outcome.

Sources: crates/yantrikdb-core/src/base/types.rs

Resolution Logic

The coherence checking system applies evidence-based resolution when two beliefs contradict each other. The algorithm compares the evidence count between the two conflicting nodes:

let (loser, winner_label) = match (node_a, node_b) {
    (Some(a), Some(b)) => {
        // Prefer keeping the one with more evidence.
        if a.attrs.evidence_count >= b.attrs.evidence_count {
            (b.id, a.label.clone())
        } else {
            (a.id, b.label.clone())
        }
    }
    (None, Some(_)) => (contradiction.belief_a, "unknown".to_string()),
    (Some(_), None) => (contradiction.belief_b, "unknown".to_string()),
    (None, None) => (contradiction.belief_a, "unknown".to_string()),
};

The losing belief is demoted (tombstoned), and the explanation reflects which belief had higher evidence support. This approach ensures that beliefs with more corroborating evidence are preserved in the knowledge graph.

Sources: crates/yantrikdb-core/src/cognition/coherence.rs

Conflict Surfacing

Not all conflicts require immediate user attention. The surfacing system uses the SurfaceReason enum to determine when and how conflicts should be presented to the user.

Surfacing Reasons Related to Conflicts

Sources: crates/yantrikdb-core/src/cognition/surfacing.rs

Surfacing Modes

When a conflict is surfaced, the system selects an appropriate presentation mode based on urgency and priority:

Sources: crates/yantrikdb-core/src/cognition/surfacing.rs

Cognitive Edge Kinds and Conflict Detection

The belief network uses typed edges to represent relationships between cognitive nodes. Certain edge types are directly relevant to conflict detection.

Epistemic Edges

Edges that participate in belief revision and conflict detection:

The system identifies conflicts when a Contradicts edge exists between two belief nodes. These edges have negative activation transfer, meaning they inhibit the target node's activation level.

Sources: crates/yantrikdb-core/src/cognition/state.rs

Edge Classification Methods

/// Whether this edge participates in belief revision.
pub fn is_epistemic(self) -> bool {
    matches!(self, Self::Supports | Self::Contradicts)
}

/// Whether this edge type is inhibitory (suppresses target activation).
pub fn is_inhibitory(self) -> bool {
    self.activation_transfer() < 0.0
}

Coherence Checking Pipeline

The coherence checking system is responsible for detecting conflicts as part of the cognitive processing pipeline.

Operator Priority in Cognitive Loop

The CoherenceCheck operator has the lowest priority, meaning it may be skipped when computational budget is constrained. This design ensures that core cognitive functions (attention, recall, belief integration) always execute first.

Sources: crates/yantrikdb-core/src/cognition/query_dsl.rs

Fragmentation Detection

The coherence system also monitors attention fragmentation—how evenly distributed activation is across working set nodes:

fn compute_fragmentation(ws: &WorkingSet) -> f64 {
    if ws.len() <= 1 {
        return 0.0;
    }
    
    let activations: Vec<f64> = ws.iter().map(|n| n.attrs.activation).collect();
    let total: f64 = activations.iter().sum();
    
    if total <= 0.0 {
        return 0.0;
    }
    
    // Normalized entropy (Shannon entropy / max entropy)
    let n = activations.len() as f64;
    // ...
}

High fragmentation (many nodes with similar activation) can indicate unresolved conflicts competing for attention.

Sources: crates/yantrikdb-core/src/cognition/coherence.rs

Python API

The Python bindings expose conflict resolution through the PyConflictEngine interface.

Key Methods

Sources: crates/yantrikdb-python/src/py_engine/cognition.rs

Resolution Example

result = db.resolve_conflict(
    conflict_id="conflict_123",
    strategy="evidence_based",
    winner_rid="memory_456",
    new_text=None,
    resolution_note="Preferred belief with higher evidence count"
)

System Architecture

graph TD
    subgraph "Cognitive Engine"
        A[CognitiveNode Storage] --> B[CoherenceChecker]
        B --> C[ConflictDetector]
        C --> D[Conflict Queue]
        D --> E[SurfaceReason Evaluator]
        E --> F[ProactiveSuggestion Generator]
    end
    
    subgraph "Conflict Types"
        G[IdentityFact]
        H[Preference]
        I[Temporal]
        J[Consolidation]
        K[Minor]
    end
    
    subgraph "Resolution Outcomes"
        L[Winner Selected]
        M[Loser Tombstoned]
        N[New Merged Memory]
        O[User Notified]
    end
    
    C --> G
    C --> H
    C --> I
    C --> J
    C --> K
    
    F --> L
    F --> M
    F --> N
    F --> O

Provenance and Reliability

Conflicts are detected based on the provenance source of each memory node. Different provenance types have different reliability priors:

When conflicts involve memories with different provenance sources, the system considers reliability priors in its resolution strategy.

Sources: crates/yantrikdb-core/src/cognition/state.rs

Summary

The Conflict Detection and Resolution system in yantrikdb provides a robust mechanism for maintaining cognitive consistency:

1. Detection: Conflicts are identified through the Contradicts edge type in the belief network and during coherence checking operations.

1. Classification: Conflicts are categorized into five types with associated priority levels, enabling appropriate handling based on severity.

1. Resolution: Evidence-based resolution selects the belief with higher evidence count as the winner, with the losing belief being tombstoned.

1. Surfacing: High-priority conflicts trigger the surfacing system to proactively notify the user through appropriate channels (whisper, nudge, alert, or preempt).

1. Integration: The system integrates with the broader cognitive engine, respecting operator priorities and computational budgets.

Related Pages

Related topics: Python Bindings

MCP Server Integration

The MCP (Model Context Protocol) Server Integration provides a standardized interface for AI agents to interact with YantrikDB's persistent cognitive memory. This integration enables AI assistants—including Claude Code, Cursor, Windsurf, and any MCP-compatible client—to automatically remember decisions, recall relevant context, and detect contradictions without explicit user prompting.

Overview

The MCP server is a Python-based component built on top of the FastMCP framework that exposes YantrikDB's core capabilities through the Model Context Protocol. This allows AI agents to maintain persistent memory across sessions, automatically consolidating experiences over time.

Sources: MCP_REDESIGN.md:1-20

Core Objectives

The MCP integration was designed to achieve the following success criteria:

Sources: MCP_REDESIGN.md:50-58

Architecture

Component Structure

The MCP server is organized into three primary modules within src/yantrikdb/mcp/:

graph TD
    A[MCP Client<br/>Claude Code, Cursor] --> B[server.py<br/>FastMCP Lifespan]
    B --> C[tools.py<br/>10 Tool Definitions]
    B --> D[resources.py<br/>MCP Resources]
    C --> E[YantrikDB Core<br/>Rust Engine]
    D --> E
    E --> F[SQLite Database<br/>memory.db]

Sources: MCP_REDESIGN.md:25-35

File Structure

Sources: MCP_REDESIGN.md:20-30

Available Tools

The MCP server exposes 10 core tools that AI agents can invoke. Each tool is designed with rich descriptions that guide auto-pilot behavior.

Sources: MCP_REDESIGN.md:31-35

Tool Reference

Sources: MCP_REDESIGN.md:32-35

Tool Description Quality

The tool descriptions are designed to be comprehensive, telling agents not just what each tool does but *when* to call it. This guidance supports auto-pilot behavior, similar to the instruction blocks in Claude Code's configuration.

Sources: MCP_REDESIGN.md:40-42

Configuration

Environment Variables

The MCP server accepts configuration through environment variables:

Sources: MCP_REDESIGN.md:36-38

Installation

pip install yantrikdb[mcp]

After installation, the server can be started with:

yantrikdb-mcp

MCP Client Configuration

Add the following to your MCP client configuration (e.g., mcp.json):

{
  "mcpServers": {
    "yantrikdb": {
      "command": "yantrikdb-mcp",
      "env": {
        "YANTRIKDB_DB_PATH": "/path/to/memory.db"
      }
    }
  }
}

Advanced Capabilities

Beyond the basic remember/recall flow, the MCP server exposes advanced YantrikDB capabilities that are available from the Rust engine but not yet fully utilized by the agent workflow.

Sources: MCP_REDESIGN.md:60-70

RecallQuery Builder Options

The underlying Rust engine supports rich query building:

Sources: crates/yantrikdb-core/src/cognition/query_dsl.rs:1-20

Conflict Resolution

The engine supports multiple conflict resolution strategies:

Sources: MCP_REDESIGN.md:62-65

Pattern Mining

Pattern mining can be configured with:

• Custom confidence thresholds
• Domain-specific pattern detection
• Temporal pattern analysis
• Entity relationship patterns

Sources: MCP_REDESIGN.md:63-65

Personality Profile Extraction

The engine can extract personality profiles from memory interactions, enabling more personalized agent behavior over time.

Sources: MCP_REDESIGN.md:64-66

Spaced Repetition Reinforcement

Memory access automatically triggers spaced repetition reinforcement, strengthening frequently accessed memories and allowing less-used ones to decay naturally.

Sources: MCP_REDESIGN.md:65-67

Batch Operations

The Python bindings support batch record operations for efficiency:

db = yantrikdb.YantrikDB.with_default("memory.db")
db.record_batch([
    {"text": "Memory 1", "importance": 0.8},
    {"text": "Memory 2", "importance": 0.6},
])

Replication and Sync

YantrikDB supports CRDT-based replication for multi-device synchronization:

ops = db.extract_ops_since(since_hlc=hlc, since_op_id=op_id)
db.apply_ops(ops)

Sources: crates/yantrikdb-python/src/py_engine/sync.rs:1-30

Memory Types

The system supports four primary memory types, each serving distinct cognitive purposes:

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:1-30

Entity and Belief Management

Entity Graph

The entity graph maintains relationships between extracted entities:

graph LR
    A[Alice] -->|leads| B[Engineering]
    B -->|part_of| C[Company]
    A -->|works_with| D[Bob]

Belief System

Beliefs have provenance types indicating their source reliability:

Sources: crates/yantrikdb-core/src/cognition/state.rs:1-50

Workflow Examples

Basic Memory Storage and Retrieval

# Using Python library directly
import yantrikdb

db = yantrikdb.YantrikDB.with_default("memory.db")

# Store a memory
db.record("Alice is the engineering lead", importance=0.8, domain="people")

# Retrieve relevant memories
results = db.recall("who leads the team?", top_k=3)

# Create a relationship
db.relate("Alice", "Engineering", "leads")

Triggering Cognitive Processing

# Run the think() cognition loop
db.think()  # consolidate, detect conflicts, mine patterns

This single call triggers:

• Memory consolidation
• Conflict detection and resolution
• Pattern mining

Sources: README.md:1-40

Cognitive Triggers

The system supports multiple trigger types for proactive memory maintenance:

Sources: crates/yantrikdb-core/src/base/types.rs:1-50

Future Enhancements

The MCP_REDESIGN.md outlines planned improvements:

1. Rich tool descriptions — More detailed examples for auto-pilot behavior
2. Server instructions — System prompt injection for agent guidance
3. Better error messages — More informative feedback for debugging
4. Streaming responses — For long-running operations
5. Progress indicators — Real-time feedback during consolidation

Sources: MCP_REDESIGN.md:45-55

See Also

• YantrikDB Core Engine
• Python API Reference
• Conflict Resolution
• Consolidation System

Related Pages

Related topics: MCP Server Integration, Installation, Core API Reference

Python Bindings

Overview

The Python bindings provide a native Python interface to yantrikdb, enabling Python developers to interact with the memory database using familiar Python idioms. Built on top of the Rust core using pyo3, the bindings expose the full functionality of yantrikdb while maintaining Pythonic conventions for parameter ordering and default values.

The PyYantrikDB class serves as the primary entry point, offering methods for recording memories, querying with semantic search, managing relationships between entities, and triggering cognitive consolidation processes.

Architecture

graph TD
    A[Python Application] --> B[PyYantrikDB]
    B --> C[pyo3 Bridge Layer]
    C --> D[yantrikdb-core]
    D --> E[SQLite Storage]
    D --> F[Vector Index]
    
    G[py_types.rs] --> C
    G --> H[Type Conversions]
    H --> I[PyObject ↔ Rust Structs]
    
    J[Default Embedder] --> B
    J --> K[potion-base-2M<br/>dim=64]

The binding layer consists of three main components:

Core API Methods

Recording Memories

The record() method stores new memories in the database with semantic embeddings. It accepts text input and generates embeddings automatically using the bundled embedder, or accepts pre-computed embeddings for efficiency.

db.record(
    text="Alice is the engineering lead",
    memory_type="episodic",
    importance=0.8,
    valence=0.0,
    half_life=604800.0,
    certainty=0.8,
    domain="people",
    source="user",
    namespace="default",
    emotional_state=None
)

Parameters:

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:13-30

Querying and Recall

The recall() method performs semantic search over stored memories, returning results ranked by relevance. It supports both text queries and pre-computed embedding vectors.

results = db.recall(
    query="who leads the team?",
    top_k=10,
    memory_type=None,
    namespace=None,
    time_window=None,
    include_consolidated=False,
    expand_entities=True,
    skip_reinforce=False,
    domain=None,
    source=None
)

Parameters:

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:60-85

The recall_text() method provides a simplified interface for text-based queries with optional filtering:

results = db.recall_text(
    query="who leads the team?",
    top_k=10,
    namespace=None,
    domain=None,
    source=None
)

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:125-145

Procedural Memory

Procedural memory stores task-related information and supports reinforcement learning for effectiveness tracking.

# Record a procedural memory
rid = db.record_procedural(
    text="How to deploy to production",
    domain="devops",
    task_context="deployment workflow",
    effectiveness=0.5,
    namespace="default"
)

# Reinforce based on outcome
db.reinforce_procedural(rid, outcome=0.9)

Parameters for record_procedural:

Sources: crates/yantrikdb-python/src/py_engine/session_temporal.rs:45-60

Memory Correction

The correct() method allows updating existing memories with corrections, maintaining an audit trail of original content.

result = db.correct(
    rid="existing-memory-rid",
    new_text="Updated information",
    new_importance=0.9,
    new_valence=0.2,
    embedding=None,
    correction_note="Corrected factual error"
)

Return Value:

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:100-115

Memory Decay

The decay() method triggers decay calculations across all memories based on access patterns and half-life values.

decayed = db.decay(threshold=0.01)

Parameters:

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:88-95

Default Embedding Model

The Python bindings include a bundled embedder (potion-base-2M) that provides 64-dimensional embeddings out of the box. This eliminates dependencies on external services like sentence-transformers or ONNX runtime.

graph LR
    A[Input Text] --> B[pyo3 embed_text]
    B --> C[potion-base-2M]
    C --> D[64-dim Vector]
    D --> E[Storage/Recall]

The embedder is invoked automatically when embedding parameters are omitted:

# Auto-embedding
db.record("Alice is the engineering lead")

# Manual embedding
vector = [0.1, 0.2, ...]  # 64 floats
db.record("Alice is the engineering lead", embedding=vector)

Sources: README.md

Initialization and Configuration

Creating a Database Instance

import yantrikdb

# Default instance with bundled embedder
db = yantrikdb.YantrikDB.with_default("memory.db")

# Work with the database
db.record("Memory content", importance=0.8)
results = db.recall("Query text")

# Always close when done
db.close()

Sources: README.md

Type Conversions

The py_types.rs module handles bidirectional conversion between Rust structs and Python objects:

Sources: crates/yantrikdb-python/src/py_types.rs:6-40

Memory to Dictionary

The memory_to_dict() function converts a core Memory struct to a Python dictionary matching the Python engine's expected output format:

pub fn memory_to_dict(py: Python<'_>, mem: &yantrikdb_core::Memory) -> PyResult<PyObject> {
    let dict = PyDict::new(py);
    dict.set_item("rid", &mem.rid)?;
    dict.set_item("type", &mem.memory_type)?;
    dict.set_item("text", &mem.text)?;
    dict.set_item("created_at", mem.created_at)?;
    dict.set_item("importance", mem.importance)?;
    // ... additional fields
    Ok(dict.into())
}

Sources: crates/yantrikdb-python/src/py_types.rs:8-25

Return Value Structure

Recall Results

Query results are returned as Python dictionaries with the following structure:

{
    "rid": "memory-unique-id",
    "type": "episodic",
    "text": "Memory content",
    "score": 0.95,           # Relevance score
    "created_at": 1234567890.0,
    "importance": 0.8,
    "valence": 0.0,
    "half_life": 604800.0,
    "last_access": 1234567890.0,
    "access_count": 5,
    "consolidation_status": "stable",
    "storage_tier": "hot",
    "namespace": "default",
    "certainty": 0.8,
    "domain": "people",
    "source": "user",
    "emotional_state": None,
    "metadata": {}
}

Advanced Query Options

Recall with Sequence Verification

For applications requiring strong consistency guarantees, recall_with_seq() ensures query results reflect all prior writes:

# After a write operation
db.record("New memory", namespace="work")

# Ensure subsequent recall sees the write
results = db.recall_with_seq(
    query_embedding=embedding,
    top_k=10,
    min_seq=prior_sequence,
    namespace="work",
    timeout=timedelta(seconds=5)
)

Sources: crates/yantrikdb-core/src/engine/recall.rs:50-80

Time-Window Filtering

Results can be filtered to a specific time range using Unix timestamps:

import time

now = time.time()
week_ago = now - 604800  # 7 days

results = db.recall(
    query="meetings",
    time_window=(week_ago, now)
)

Memory Types

yantrikdb supports multiple memory types for different kinds of information:

Sources: crates/yantrikdb-core/src/cognition/state.rs:120-145

Relationship Management

Beyond storing individual memories, yantrikdb supports graph-like relationships between entities:

# Define relationships
db.relate("Alice", "Engineering", "leads")
db.relate("Alice", "Bob", "manages")

# Query relationships
edges = db.get_edges("Alice")

The system supports relationship types including:

Sources: crates/yantrikdb-core/src/cognition/state.rs:200-230

Cognitive Processing

Think Operation

The think() method triggers the cognitive processing pipeline:

db.think()  # Consolidate, detect conflicts, mine patterns

This operation:

1. Consolidates related memories
2. Detects conflicts between beliefs
3. Mines patterns from episodic data
4. Updates procedural memory effectiveness

Error Handling

The Python bindings map Rust errors to appropriate Python exceptions:

try:
    db.record("Memory")
except RuntimeError as e:
    print(f"Database error: {e}")
except ValueError as e:
    print(f"Invalid input: {e}")

Sources: crates/yantrikdb-python/src/py_engine/memory.rs:20-25

Best Practices

``python db = yantrikdb.YantrikDB("memory.db") try: # operations finally: db.close() ``

1. Always close the database when done to ensure proper cleanup:

1. Use context managers when possible for automatic cleanup

1. Batch operations when recording multiple related memories

1. Choose appropriate namespaces to partition data logically

1. Set importance values appropriately to control memory retention and retrieval priority

Doramagic Pitfall Log

Doramagic extracted 16 source-linked risk signals. Review them before installing or handing real data to the project.

1. Installation risk: API addition: deterministic mutation primitives (record_with_rid + friends) for cluster-mode replication

• Severity: medium
• Finding: Installation risk is backed by a source signal: API addition: deterministic mutation primitives (record_with_rid + friends) for cluster-mode replication. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/issues/9

2. Installation risk: Bug: `namespace` parameter ignored in batch `remember` calls — memories always stored under `default`

• Severity: medium
• Finding: Installation risk is backed by a source signal: Bug: namespace parameter ignored in batch remember calls — memories always stored under default. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/issues/2

3. Installation risk: Migration v14→v15 fails: ALTER TABLE on edges view

• Severity: medium
• Finding: Installation risk is backed by a source signal: Migration v14→v15 fails: ALTER TABLE on edges view. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/issues/10

4. Installation risk: [bug] Tombstoned memories still appear in similarity-scan recall results

• Severity: medium
• Finding: Installation risk is backed by a source signal: [bug] Tombstoned memories still appear in similarity-scan recall results. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/issues/8

5. Installation risk: [bug] YANTRIKDB_ENCRYPTION_KEY_HEX env var ignored — encryption silently disabled

• Severity: medium
• Finding: Installation risk is backed by a source signal: [bug] YANTRIKDB_ENCRYPTION_KEY_HEX env var ignored — encryption silently disabled. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/issues/6

6. Installation risk: v0.7.10 — Fix has_embedder() for Python-side embedders (plugin#4)

• Severity: medium
• Finding: Installation risk is backed by a source signal: v0.7.10 — Fix has_embedder() for Python-side embedders (plugin#4). Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/releases/tag/v0.7.10

7. Installation risk: v0.7.11 — pyo3 0.28.3 + python3.14 Support

• Severity: medium
• Finding: Installation risk is backed by a source signal: v0.7.11 — pyo3 0.28.3 + python3.14 Support. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/releases/tag/v0.7.11

8. Installation risk: v0.7.4 — Python Bindings: with_default + record_text/recall_text

• Severity: medium
• Finding: Installation risk is backed by a source signal: v0.7.4 — Python Bindings: with_default + record_text/recall_text. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/releases/tag/v0.7.4

9. Installation risk: v0.7.5 — Python UX: TypeError Guard + embedder-download in Default Wheel

• Severity: medium
• Finding: Installation risk is backed by a source signal: v0.7.5 — Python UX: TypeError Guard + embedder-download in Default Wheel. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/releases/tag/v0.7.5

10. Configuration risk: Configuration risk needs validation

• Severity: medium
• Finding: Configuration risk is backed by a source signal: Configuration risk needs validation. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: capability.host_targets | github_repo:1164482810 | https://github.com/yantrikos/yantrikdb | host_targets=mcp_host, claude, claude_code

11. Configuration risk: v0.7.7 — recall_text Keyword-Only Filter Args

• Severity: medium
• Finding: Configuration risk is backed by a source signal: v0.7.7 — recall_text Keyword-Only Filter Args. Treat it as a review item until the current version is checked.
• User impact: Users may get misleading failures or incomplete behavior unless configuration is checked carefully.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/yantrikos/yantrikdb/releases/tag/v0.7.7

12. Capability assumption: README/documentation is current enough for a first validation pass.

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: The project should not be treated as fully validated until this signal is reviewed.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: capability.assumptions | github_repo:1164482810 | https://github.com/yantrikos/yantrikdb | README/documentation is current enough for a first validation pass.

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using yantrikdb with real data or production workflows.

• API addition: deterministic mutation primitives (record_with_rid + frien - github / github_issue
• Migration v14→v15 fails: ALTER TABLE on edges view - github / github_issue
• [[bug] Tombstoned memories still appear in similarity-scan recall results](https://github.com/yantrikos/yantrikdb/issues/8) - github / github_issue
• [[bug] POST /v1/admin/snapshot unusable in single-node mode — requires cl](https://github.com/yantrikos/yantrikdb/issues/7) - github / github_issue
• [[bug] YANTRIKDB_ENCRYPTION_KEY_HEX env var ignored — encryption silently](https://github.com/yantrikos/yantrikdb/issues/6) - github / github_issue
• [[bug] at-rest encryption key_hex in TOML has no effect on disk (v0.5.0](https://github.com/yantrikos/yantrikdb/issues/3) - github / github_issue
• Bug: namespace parameter ignored in batch remember calls — memories - github / github_issue
• think() runs consolidation before conflict detection — contradictions ge - github / github_issue
• v0.7.11 — pyo3 0.28.3 + python3.14 Support - github / github_release
• v0.7.10 — Fix has_embedder() for Python-side embedders (plugin#4) - github / github_release
• v0.7.9 — Bundle potion-multilingual-128M (101 Languages) in embedder-dow - github / github_release
• v0.7.8 — Extended Idempotent Migration Runner (closes #10) - github / github_release