Related Pages

Related topics: System Architecture, Getting Started

Introduction to txtai

txtai is an all-in-one AI framework for semantic search, large language model (LLM) orchestration, and language model workflows. It provides a unified platform that combines vector search capabilities with traditional database features, enabling developers to build sophisticated AI-powered applications without managing multiple disparate systems.

Overview

The central innovation of txtai is its embeddings database - a hybrid data store that unifies:

• Sparse vector indexes for keyword-based search
• Dense vector indexes for semantic similarity
• Graph networks for relationship modeling
• Relational databases for structured data storage

This architecture enables txtai to function both as a standalone vector search engine and as a powerful knowledge source for RAG (Retrieval Augmented Generation) applications. Sources: README.md:1-20

Architecture

The txtai architecture follows a layered design that separates concerns while maintaining tight integration between components.

graph TD
    subgraph "txtai Architecture"
        A[Application Layer] --> B[Workflow Engine]
        B --> C[Pipeline System]
        C --> D[Embeddings Database]
        D --> E[(Vector Index)]
        D --> F[(Graph Network)]
        D --> G[(Relational DB)]
    end
    
    subgraph "Pipelines"
        H[LLM Pipeline] 
        I[Data Pipeline]
        J[Text Pipeline]
        K[Audio Pipeline]
    end
    
    C --> H
    C --> I
    C --> J
    C --> K

Core Components

Key Features

Vector Search Capabilities

txtai provides comprehensive vector search functionality including:

• Semantic/similarity/vector/neural search Sources: README.md:45-50
• SQL integration for hybrid queries
• Object storage support
• Topic modeling
• Graph analysis
• Multimodal indexing (text, audio, images, video)

Pipeline System

The pipeline system enables flexible model execution:

Workflow Engine

Workflows orchestrate complex AI tasks using a template-based system:

graph LR
    A[Input] --> B[Template Task]
    B --> C[LLM Pipeline]
    C --> D[Output]
    
    B -.->|config| E[Template Rules]
    E -->|format| B

The template system supports named parameters, positional arguments, and strict/conservative parsing modes. Sources: src/python/txtai/workflow/task/template.py:1-40

Installation

Core Dependencies

# Default installation includes
default = [
    "faiss-cpu>=1.7.1.post2",
    "huggingface-hub>=0.34.0",
    "msgpack>=1.0.7",
    "numpy>=1.18.4",
    "regex>=2022.8.17",
    "pyyaml>=5.3",
    "safetensors>=0.4.5",
    "torch>=2.4",
    "transformers>=4.56.2",
]

Sources: setup.py:18-30

Optional Installations

Sources: setup.py:55-150

Basic Usage

Semantic Search

import txtai

embeddings = txtai.Embeddings()
embeddings.index(["Correct", "Not what we hoped"])
result = embeddings.search("positive", 1)
# Returns: [(0, 0.29862046241760254)]

Sources: README.md:35-42

Text Extraction

from txtai.pipeline import Textractor

textractor = Textractor(backend="docling", sections=True)
for chunk in textractor("document.pdf"):
    process(chunk)

Sources: examples/rag_quickstart.py:1-50

RAG Pipeline

from txtai import Embeddings, RAG
from txtai.pipeline import Textractor

# Build embeddings database
embeddings = Embeddings(content=True, path="Qwen/Qwen3-Embedding-0.6B")
embeddings.index(chunks)

# Create RAG pipeline
template = """
  Answer the following question using the provided context.
  Question: {question}
  Context: {context}
"""

rag = RAG(
    embeddings,
    "Qwen/Qwen3-0.6B",
    system="You are a friendly assistant",
    template=template,
)

result = rag("Summarize the main advancements made by BERT")

Sources: examples/rag_quickstart.py:50-80

Use Cases

Semantic Search

Traditional keyword-based search systems match exact terms. Semantic search understands natural language and identifies results with similar meaning, regardless of keyword overlap. Sources: README.md:45-55

RAG Applications

The embeddings database serves as a knowledge source for LLM applications:

graph TD
    A[User Query] --> B[Embeddings Search]
    B --> C[Retrieve Context]
    C --> D[LLM Generation]
    D --> E[Response]
    
    F[(Embeddings DB)] -.->|retrieval| C

Autonomous Agents

Agents utilize tools to perform complex tasks:

from txtai.agent.tool.read import ReadTool

tool = ReadTool(maxlength=40000)
content = tool.forward("path/to/file.txt")

Sources: src/python/txtai/agent/tool/read.py:1-60

API Server

txtai includes a built-in REST API for cross-language compatibility:

# app.yml
embeddings:
    path: sentence-transformers/all-MiniLM-L6-v2

CONFIG=app.yml uvicorn "txtai.api:app"
curl -X GET "http://localhost:8000/search?query=positive"

Sources: README.md:50-60

Design Principles

1. Low footprint - Install additional dependencies and scale up when needed Sources: README.md:60-70
2. Local execution - No need to ship data to remote services
3. Model flexibility - Work with micromodels up to large language models
4. Modular design - Use individual components or the full stack

Requirements

• Python 3.10+
• PyTorch 2.4+
• Transformers 4.56.2+

The framework supports Hugging Face models, llama.cpp, Ollama, vLLM, and more for embeddings and LLM operations. Sources: setup.py:18-30

Related Pages

Related topics: Introduction to txtai

Getting Started

txtai is an all-in-one AI framework for semantic search, LLM orchestration and language model workflows. The Getting Started guide provides the essential knowledge needed to install, configure, and begin using txtai effectively.

Overview

txtai provides a unified interface for building AI applications with:

• Vector search with SQL, object storage, topic modeling, graph analysis and multimodal indexing
• Embeddings for text, documents, audio, images and video
• Pipelines powered by language models for prompts, question-answering, labeling, transcription, and translation
• Workflows that orchestrate multi-step processes
• Agents for autonomous decision-making

Sources: README.md

Installation

Requirements

Sources: setup.py:27-37

Installation Methods

#### Standard Installation

Standard installation includes all default dependencies:

pip install txtai

This installs the core package with default dependencies including FAISS, HuggingFace Hub, msgpack, numpy, regex, PyYAML, safetensors, PyTorch, and transformers.

#### Minimal Installation

For environments with limited resources, install a minimal version:

MINIMAL=1 pip install txtai

Sources: setup.py:18-25

Optional Extras

txtai provides modular extras to scale functionality based on needs:

Sources: setup.py:38-98

Install with extras:

pip install txtai[api,workflow]
pip install txtai[all]

Core Concepts

Embeddings Database

The central component of txtai is the embeddings database, which combines:

• Vector indexes (sparse and dense)
• Graph networks
• Relational databases

This foundation enables vector search and serves as a knowledge source for LLM applications.

Sources: README.md

graph TD
    A[Embeddings Database] --> B[Vector Indexes]
    A --> C[Graph Networks]
    A --> D[Relational Database]
    B --> E[Dense Vectors]
    B --> F[Sparse Vectors]

Pipeline Architecture

Pipelines are language model powered components that perform specific tasks:

graph LR
    A[Input Data] --> B[Pipeline]
    B --> C[LLM Pipeline]
    B --> D[Data Pipeline]
    B --> E[Text Pipeline]
    C --> F[Generated Content]
    D --> G[Extracted Data]
    E --> H[Processed Text]

Key pipeline types:

Sources: src/python/txtai/pipeline/llm/llm.py

Workflow System

Workflows orchestrate multi-step processes by connecting tasks:

graph TD
    A[Workflow Input] --> B[Task 1]
    B --> C[Task 2]
    C --> D[Task N]
    D --> E[Workflow Output]
    
    F[Template Task] -->|formats| B
    G[URL Task] -->|fetches| B

Sources: examples/workflows.py

Quick Start

Basic Semantic Search

The simplest way to get started with txtai is semantic search:

import txtai

embeddings = txtai.Embeddings()
embeddings.index(["Correct", "Not what we hoped"])
embeddings.search("positive", 1)
# Returns: [(0, 0.29862046241760254)]

Sources: README.md

Building a RAG Application

Retrieval Augmented Generation combines embeddings search with LLM inference:

from txtai import Embeddings, RAG
from txtai.pipeline import Textractor

# Step 1: Extract text from documents
textractor = Textractor()
chunks = []
for f in ["document1.pdf", "document2.pdf"]:
    for chunk in textractor(f):
        chunks.append((f, chunk))

# Step 2: Build embeddings database
embeddings = Embeddings(content=True, path="Qwen/Qwen3-Embedding-0.6B", maxlength=2048)
embeddings.index(chunks)

# Step 3: Create RAG pipeline
template = """
  Answer the following question using the provided context.

  Question:
  {question}

  Context:
  {context}
"""

rag = RAG(
    embeddings,
    "Qwen/Qwen3-0.6B",
    system="You are a friendly assistant",
    template=template,
    output="flatten",
)

# Step 4: Query
question = "Summarize the main advancements made by BERT"
print(rag(question, maxlength=2048, stripthink=True))

Sources: examples/rag_quickstart.py

Wikipedia Summarization

Example combining external API with summarization:

import requests
from txtai.pipeline import Summary

class Application:
    SEARCH_TEMPLATE = "https://en.wikipedia.org/w/api.php?action=opensearch&search=%s&limit=1&namespace=0&format=json"
    CONTENT_TEMPLATE = "https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles=%s"

    def __init__(self):
        self.summary = Summary("sshleifer/distilbart-cnn-12-6")

    def query(self, query):
        # Search Wikipedia
        data = requests.get(self.SEARCH_TEMPLATE % query).json()
        if data and data[1]:
            page = data[1][0]
            content = requests.get(self.CONTENT_TEMPLATE % page).json()
            content = list(content["query"]["pages"].values())[0]["extract"]
            return self.summary(content)
        return None

Sources: examples/wiki.py

Configuration

Embeddings Configuration

Sources: examples/rag_quickstart.py

LLM Parameters

Sources: src/python/txtai/pipeline/llm/llm.py:18-34

RAG Configuration

API Service

txtai includes a built-in API for language-agnostic applications:

# app.yml
embeddings:
    path: sentence-transformers/all-MiniLM-L6-v2

CONFIG=app.yml uvicorn "txtai.api:app"
curl -X GET "http://localhost:8000/search?query=positive"

Benefits of the API service:

• Work with your programming language of choice
• Run local - no need to ship data to remote services
• Supports models from micromodels to large language models
• Low footprint - scale up when needed

Sources: README.md

Next Steps

After completing the Getting Started guide, explore:

1. Example Notebooks - Over 70 example notebooks covering all functionality
2. Workflows - Build complex multi-step pipelines
3. Agents - Create autonomous AI agents with tool use
4. Graph Analysis - Network and relationship analysis
5. Model Training - Fine-tune models for your use case

All examples are available at: https://neuml.github.io/txtai/examples

Related Pages

Related topics: Embeddings Database, Database Integration

System Architecture

txtai is an all-in-one AI framework designed for semantic search, LLM orchestration, and language model workflows. The architecture is built around a modular, extensible design that allows developers to scale from micromodels to large language models with minimal footprint.

Architecture Overview

The txtai architecture consists of several interconnected layers that work together to provide comprehensive AI capabilities.

graph TB
    subgraph "Client Layer"
        API["REST API"]
        Console["Console"]
        Library["Python Library"]
    end
    
    subgraph "Core Engine"
        Embeddings["Embeddings Database"]
        Workflow["Workflow Engine"]
        Agent["Agent System"]
    end
    
    subgraph "Pipeline Layer"
        LLMPipeline["LLM Pipeline"]
        DataPipeline["Data Pipeline"]
        TextPipeline["Text Pipeline"]
        TrainPipeline["Train Pipeline"]
        AudioPipeline["Audio Pipeline"]
        ImagePipeline["Image Pipeline"]
    end
    
    subgraph "Index Layer"
        ANN["ANN Index"]
        Documents["Document Store"]
        Graph["Graph Network"]
        Database["Relational DB"]
    end
    
    API --> Embeddings
    API --> Workflow
    API --> Agent
    Embeddings --> ANN
    Embeddings --> Documents
    Embeddings --> Graph
    Embeddings --> Database

Sources: README.md

Core Components

Embeddings Database

The embeddings database is the foundational component of txtai. It serves as a union of multiple index types:

Sources: README.md

The embeddings database enables vector search capabilities and serves as a powerful knowledge source for large language model (LLM) applications.

Pipeline System

Pipelines are the processing units of txtai, each designed for specific tasks:

graph LR
    Input["Input Data"] --> Pipeline["Pipeline"]
    
    subgraph "Pipeline Types"
        Audio["Audio Pipeline"]
        Data["Data Pipeline"]
        Image["Image Pipeline"]
        LLM["LLM Pipeline"]
        Text["Text Pipeline"]
        Train["Train Pipeline"]
    end
    
    Pipeline --> Output["Output Data"]

#### Data Pipeline

The Data Pipeline handles text extraction and processing:

The Textractor class supports multiple backends for file extraction:

def __init__(
    self,
    sentences=False,
    lines=False,
    paragraphs=False,
    minlength=None,
    join=False,
    sections=False,
    cleantext=True,
    chunker=None,
    headers=None,
    backend="available",
    safeopen=False,
    **kwargs,
):

Sources: src/python/txtai/pipeline/data/textractor.py:19-37

#### LLM Pipeline

The LLM Pipeline provides LLM generation capabilities:

def __call__(
    self,
    text,
    maxlength=None,
    stream=False,
    stop=None,
    defaultrole="auto",
    stripthink=None,
    **kwargs,
):

Key features:

• Supports streaming responses
• Configurable stop sequences
• Built-in thinking tag stripping
• Chat and vision model support

Sources: src/python/txtai/pipeline/llm/llm.py:1-50

Agent System

The Agent System enables autonomous AI agents with tool use capabilities:

graph TD
    User["User Input"] --> Agent["Agent"]
    Agent --> Tools["Tool Collection"]
    Tools --> EmbeddingsTool["Embeddings Tool"]
    Tools --> FunctionTool["Function Tool"]
    Tools --> DefaultTools["Default Tools"]
    
    subgraph "Tool Factory"
        create["create()"]
        createtool["createtool()"]
    end
    
    Tools --> Result["Result"]
    Result --> Agent

#### FunctionTool

A FunctionTool wraps descriptive configuration with a target function for LLM prompts:

class FunctionTool(Tool):
    def __init__(self, config):
        self.name = config["name"]
        self.description = config["description"]
        self.inputs = config["inputs"]
        self.output_type = config.get("output", config.get("output_type", "any"))
        self.target = config["target"]

Sources: src/python/txtai/agent/tool/function.py:1-50

#### Tool Factory

The ToolFactory creates tools from various sources:

Sources: src/python/txtai/agent/tool/factory.py:1-100

Workflow Engine

The Workflow Engine processes tasks through a series of steps:

graph LR
    Task["Task Input"] --> Template["Template Task"]
    Template --> Process["Process"]
    Process --> Output["Task Output"]

#### Template Task

The TemplateTask generates text from templates, supporting LLM prompt preparation:

class TemplateTask(Task):
    def register(self, template=None, rules=None, strict=True):
        self.template = template if template else self.defaulttemplate()
        self.rules = rules if rules else self.defaultrules()
        self.formatter = TemplateFormatter() if strict else Formatter()

Template processing supports three input types:

Sources: src/python/txtai/workflow/task/template.py:1-50

Module Organization

Package Structure

txtai/
├── agent/           # Agent system and tools
├── api/             # REST API implementation
├── cloud/           # Cloud integration (Hub)
├── embeddings/      # Embeddings database
├── scoring/         # Scoring/ranking system
├── pipeline/       # Pipeline components
│   ├── audio/      # Audio processing
│   ├── data/       # Data extraction
│   ├── image/      # Image processing
│   ├── llm/        # LLM generation
│   ├── text/       # Text processing
│   └── train/      # Model training
└── workflow/       # Workflow engine

Dependency Groups

The project uses optional dependency groups for modular installation:

Sources: setup.py

Data Flow Architecture

Semantic Search Flow

sequenceDiagram
    participant User
    participant Embeddings
    participant ANN
    participant Documents
    
    User->>Embeddings: index(documents)
    Embeddings->>ANN: build(vector)
    Embeddings->>Documents: store(data)
    
    User->>Embeddings: search(query)
    Embeddings->>ANN: query(vector)
    ANN-->>Embeddings: top-k results
    Embeddings->>Documents: retrieve(ids)
    Documents-->>Embeddings: document content
    Embeddings-->>User: ranked results

RAG Pipeline Flow

graph TD
    Question["User Question"] --> Embeddings["Embeddings DB"]
    Embeddings --> Context["Retrieved Context"]
    Context --> LLM["LLM Pipeline"]
    LLM --> Answer["Generated Answer"]
    
    subgraph "Indexing Phase"
        Documents["Documents"] --> Textractor["Textractor"]
        Textractor --> Chunks["Text Chunks"]
        Chunks --> Embeddings
    end

Cloud Integration

Hub Module

The Hub module enables model and data sharing via Hugging Face:

def upload(self, path):
    with open(path, "r", encoding="utf-8") as f:
        content = f.read()
    
    if "embeddings " not in content:
        content += "documents filter=lfs diff=lfs merge=lfs -text\n"
        content += "embeddings filter=lfs diff=lfs merge=lfs -text\n"
    
    huggingface_hub.upload_file(...)

Features:

• Automatic LFS tracking for embeddings
• Token-based authentication
• Remote index management

Sources: src/python/txtai/cloud/hub.py

Use Cases Built on Architecture

Summary

The txtai architecture provides a flexible, extensible foundation for AI applications:

1. Embeddings Database - Unified vector/graph/relational indexing
2. Pipeline System - Modular processing for diverse data types
3. Agent System - Tool-augmented autonomous AI
4. Workflow Engine - Task orchestration and automation
5. Cloud Integration - Model sharing and deployment

This modular design enables developers to start with minimal installations and scale up capabilities as needed, supporting everything from simple semantic search to complex multi-model RAG systems with autonomous agents.

Related Pages

Related topics: System Architecture, Scoring and Retrieval Algorithms

Embeddings Database

The Embeddings Database is the foundational component of txtai, providing unified vector search capabilities that combine sparse and dense vector indexing, graph networks, and relational database functionality.

Overview

The txtai embeddings database serves as a comprehensive solution for semantic search and vector-based data retrieval. It extends traditional database capabilities by incorporating neural network-based embedding generation and similarity search.

Core Architecture:

graph TD
    A[Embeddings Database] --> B[Vector Indexes]
    A --> C[Graph Networks]
    A --> D[Relational Database]
    B --> B1[Dense Vectors]
    B --> B2[Sparse Vectors]

Sources: README.md

Key Features

Sources: README.md

Installation and Setup

Basic Installation

import txtai

embeddings = txtai.Embeddings()

Configuration Options

The embeddings database can be configured with the following key parameters:

Sources: examples/rag_quickstart.py

Core Operations

Indexing Documents

embeddings = txtai.Embeddings()
embeddings.index(["Correct", "Not what we hoped"])

Sources: README.md

Semantic Search

results = embeddings.search("positive", 1)
# Returns: [(0, 0.29862046241760254)]

The search method returns tuples of (index, score) ordered by similarity.

Architecture Components

Vector Indexes

The embeddings database supports multiple vector index backends:

Sources: setup.py

Vector Models

Supported embedding model providers:

• Sentence Transformers: sentence-transformers>=5.0.0
• ONNX Models: onnx>=1.11.0, onnxruntime>=1.11.0
• llama.cpp: llama-cpp-python>=0.2.75
• LiteLLM: litellm>=1.37.16
• Model2Vec: model2vec>=0.3.0
• Static Vectors: staticvectors>=0.2.0

Sources: setup.py

Data Flow

graph LR
    A[Input Text] --> B[Tokenizer]
    B --> C[Embedding Model]
    C --> D[Vector Index]
    D --> E[Search Query]
    E --> F[Similarity Score]
    F --> G[Results]
    
    H[(Content Store)] --> G

RAG Integration

The embeddings database serves as the knowledge source for Retrieval Augmented Generation (RAG) workflows:

from txtai import Embeddings, RAG

# Build embeddings database
embeddings = Embeddings(content=True, path="Qwen/Qwen3-Embedding-0.6B", maxlength=2048)
embeddings.index(chunks)

# Create RAG pipeline
rag = RAG(
    embeddings,
    "Qwen/Qwen3-0.6B",
    system="You are a friendly assistant",
    template=template,
    output="flatten",
)

Sources: examples/rag_quickstart.py

RAG Parameters

Sources: src/python/txtai/pipeline/llm/rag.py

API Deployment

The embeddings database can be exposed as a REST API:

# app.yml
embeddings:
    path: sentence-transformers/all-MiniLM-L6-v2

CONFIG=app.yml uvicorn "txtai.api:app"
curl -X GET "http://localhost:8000/search?query=positive"

Sources: README.md

Workflow Integration

The embeddings database integrates with txtai workflows for processing pipelines:

from txtai.workflow import Workflow, Task

# Text extraction workflow
textractor = Textractor()
chunks = []
for f, chunk in textractor(files):
    chunks.append((f, chunk))

# Index extracted content
embeddings = Embeddings(content=True, path="Qwen/Qwen3-Embedding-0.6B")
embeddings.index(chunks)

Sources: examples/rag_quickstart.py

Dependencies

Core Dependencies

Sources: setup.py

Optional Dependencies

Use Cases

Semantic Search

Traditional search systems use keywords to find data. Semantic search understands natural language and identifies results with the same meaning, not necessarily the same keywords.

Knowledge Base for LLMs

The embeddings database serves as a powerful knowledge source for LLM applications, enabling retrieval augmented generation (RAG) processes.

Multimodal Applications

• Text: Document embeddings for text classification and clustering
• Images: Visual similarity search
• Audio: Speech-to-text with semantic search
• Video: Frame-level semantic indexing

Sources: README.md

Performance Considerations

• Index Size: Vector indexes scale with document count and embedding dimensionality
• GPU Acceleration: Enable gpu=True for faster inference on compatible hardware
• Quantization: Use quantize=True to reduce memory footprint with minimal accuracy loss
• Batch Processing: Index documents in batches for optimal throughput

See Also

• RAG Pipeline - Retrieval Augmented Generation
• Similarity Pipeline - Text similarity scoring
• Workflows - Pipeline orchestration
• Agent Framework - Autonomous AI agents

Related Pages

Related topics: Workflows, Agents

Pipelines

Overview

Pipelines are the core processing components in txtai that power language model workflows. They provide a unified interface for various text processing, transformation, and generation tasks including text extraction, summarization, transcription, translation, and LLM inference.

The pipeline system enables txtai to work with text, documents, audio, images and video by providing modular, composable processing units that can be chained together to build complex workflows. Sources: setup.py

Architecture

graph TD
    subgraph "Pipeline Categories"
        Text[Text Pipelines]
        Audio[Audio Pipelines]
        LLM[LLM Pipelines]
        Data[Data Pipelines]
    end
    
    subgraph "Base Infrastructure"
        Base[Pipeline Base Class]
        Factory[Pipeline Factory]
    end
    
    Text --> Base
    Audio --> Base
    LLM --> Base
    Data --> Base
    Base --> Factory

Pipeline Categories

Text Pipelines

Text pipelines handle natural language processing tasks such as entity recognition, text classification, and language detection.

Sources: src/python/txtai/pipeline/text/__init__.py

Audio Pipelines

Audio pipelines enable speech-to-text transcription and processing of audio content.

These pipelines require scipy, sounddevice, soundfile, and webrtcvad-wheels dependencies. Sources: setup.py

LLM Pipelines

LLM (Large Language Model) pipelines provide interfaces for text generation, chat completion, and vision-capable model interactions.

graph LR
    A[Input Text/List/Dict] --> B[Generator]
    B --> C[stream]
    B --> D[batch]
    C --> E[Streaming Response]
    D --> F[Complete Response]

The LLM pipeline supports multiple backends including HuggingFace transformers and liteLLM for unified LLM access. Key capabilities include:

• Chat completion: Supports both raw prompts and structured chat templates
• Vision support: Can process image inputs with vision-capable models
• Streaming: Real-time token-by-token response streaming
• Stop sequences: Configurable stop tokens for controlled generation

Sources: src/python/txtai/pipeline/llm/llm.py, src/python/txtai/pipeline/llm/huggingface.py

Data Pipelines

Data pipelines handle document parsing, text extraction, and content conversion.

#### Textractor

The Textractor class extracts text from files using various backends:

Textractor(
    sentences=False,      # Split into sentences
    lines=False,          # Split into lines
    paragraphs=False,     # Split into paragraphs
    minlength=None,       # Minimum text length filter
    join=False,           # Join segments
    sections=False,       # Extract sections
    cleantext=True,       # Clean extracted text
    chunker=None,         # Custom text chunker
    headers=None,         # HTTP headers for URLs
    backend="available",  # "tika", "docling", or "available"
    safeopen=False        # Restrict to safe URLs only
)

Supported backends:

• Tika: Apache Tika for comprehensive document parsing
• Docling: Alternative backend using docling library

Sources: src/python/txtai/pipeline/data/textractor.py

#### FileToHTML

Converts various file formats to HTML for further processing.

graph LR
    A[File Input] --> B[Backend Selection]
    B --> C{Tika Available?}
    C -->|Yes| D[Tika Backend]
    C -->|No| E{Docling Available?}
    E -->|Yes| F[Docling Backend]
    E -->|No| G[Return None]
    D --> H[HTML Output]
    F --> H

The backend detection follows this priority:

1. Tika (if tika package is installed)
2. Docling (if docling package is installed)
3. Returns None if no backend available

Sources: src/python/txtai/pipeline/data/filetohtml.py

#### HTMLToMarkdown

Converts HTML content to Markdown format, supporting paragraph and section-level processing.

Pipeline Factory

The pipeline factory pattern enables dynamic pipeline instantiation based on configuration:

# Via factory
pipeline = PipelineFactory.create(config)

# Direct instantiation
pipeline = Summary("model-name")
pipeline = Textractor()

This allows pipelines to be defined in YAML configuration files and instantiated programmatically. Sources: src/python/txtai/pipeline/factory.py

Common Patterns

Text Processing Chain

graph TD
    A[File/URL] --> B[Textractor]
    B --> C[Segmentation]
    C --> D[Text Clean]
    D --> E[Ready for Embeddings]

LLM Generation Flow

from txtai.pipeline import LLM

# Direct model loading
llm = LLM("model-name")

# Generate with various input formats
result = llm("What is artificial intelligence?")
result = llm([{"role": "user", "content": "Hello"}])
result = llm("Generate a summary", stream=True)

Parameters supported by LLM pipelines:

Sources: src/python/txtai/pipeline/llm/llm.py

Dependencies

Pipelines are organized into optional dependency groups:

extras["pipeline-audio"]   # Audio processing
extras["pipeline-data"]     # Document parsing
extras["pipeline-image"]    # Image processing
extras["pipeline-llm"]      # LLM inference
extras["pipeline-text"]     # NLP tasks
extras["pipeline-train"]    # Model training

Full pipeline installation:

pip install txtai[pipeline]

Sources: setup.py

Integration with Workflows

Pipelines integrate seamlessly with the workflow system:

from txtai.workflow import Workflow, Task
from txtai.pipeline import Summary, Textractor

# Create a pipeline-based workflow
workflow = Workflow([
    Task(Textractor(paragraphs=True)),
    Task(Summary())
])

# Process documents
results = workflow(["path/to/document.pdf"])

The TemplateTask enables prompt template processing within workflows, supporting string formatting with named or positional parameters. Sources: src/python/txtai/workflow/task/template.py

See Also

• Workflows - Chaining pipelines together
• Embeddings - Vector indexing and search
• Agents - LLM-powered autonomous agents

Related Pages

Related topics: Pipelines

Workflows

Overview

Workflows in txtai provide a declarative, pipeline-based approach to processing data through a series of connected tasks. They enable developers to chain together multiple processing components—including text extraction, summarization, translation, and large language model (LLM) calls—into cohesive data processing pipelines.

The workflow system supports:

• Deterministic pipelines: Define processing steps that execute in sequence
• LLM-powered workflows: Integrate language model prompts for intelligent processing
• Parallel execution: Configure thread or process-based concurrency
• Flexible data transformation: Handle one-to-many and many-to-one data transformations
• Template-based prompts: Generate prompts dynamically using template formatting

Sources: examples/workflow_quickstart.py:1-25

Architecture

graph TD
    A[Input Data] --> B[Workflow]
    B --> C[Task 1: Textractor]
    C --> D[Task 2: Summary]
    D --> E[Task N: Translation/LLM]
    E --> F[Output Results]
    
    G[Task Configuration] --> B
    H[Initialize Action] --> C
    I[Finalize Action] --> E

Core Components

Sources: src/python/txtai/workflow/__init__.py:1-50

Workflow Configuration

Python API

from txtai import Workflow
from txtai.workflow import Task

workflow = Workflow([
    Task(textractor),
    Task(summary),
    Task(translate)
])

# Execute workflow
results = list(workflow(input_data))

YAML Configuration

workflow:
  tasks:
    - action: textractor
      task: textractor
    - action: summary
      task: summary

Sources: src/python/txtai/workflow/factory.py:1-30

Task System

Task Base Class

The Task class is the fundamental building block of workflows. Each task defines:

• Action: Callable function or list of functions to execute
• Select: Filter functions to select specific data
• Concurrency: Thread or process-based execution
• Data handling: Unpacking, column selection, and merging strategies

graph LR
    A[Input Element] --> B{Selection Filter}
    B -->|Pass| C[Action Execution]
    B -->|Fail| D[Skip Element]
    C --> E[One-to-Many Transform]
    E --> F[Output Elements]

#### Task Parameters

Sources: src/python/txtai/workflow/task/base.py:15-45

TemplateTask

TemplateTask extends the base Task class with template processing capabilities for generating LLM prompts.

from txtai.workflow.task import TemplateTask

def summary_template(text):
    return f"""Summarize the following text in 40 words or less.
    
{text}
"""

def translate_template(text, language):
    return f"""Translate the following text to {language}.
    
{text}
"""

workflow = Workflow([
    Task(textractor),
    Task(llm, action=lambda inputs: llm([summary_template(x) for x in inputs])),
    Task(llm, action=lambda inputs: llm([translate_template(x, "fr") for x in inputs]))
])

#### Template Processing

Sources: src/python/txtai/workflow/task/template.py:1-40

Workflow Execution

Basic Execution Flow

sequenceDiagram
    participant User
    participant Workflow
    participant Task1
    participant TaskN
    
    User->>Workflow: workflow(data)
    Workflow->>Task1: process(input)
    Task1->>Task1: select()
    Task1->>Task1: action()
    Task1->>Task1: onetomany transform
    Task1-->>TaskN: intermediate results
    TaskN->>TaskN: select()
    TaskN->>TaskN: action()
    TaskN-->>User: final results

Concurrency Modes

Sources: src/python/txtai/workflow/base.py:1-60

Pipeline Components

Available Pipeline Tasks

Quick Start Example

from txtai import LLM, Workflow
from txtai.pipeline import Summary, Textractor, Translation
from txtai.workflow import Task

# Step 1: Define available pipelines
textractor = Textractor(backend="docling", headers={"user-agent": "Mozilla/5.0"})
summary = Summary()
translate = Translation()

# Step 2: Define workflow tasks
workflow = Workflow([
    Task(textractor),           # Extract text from URL
    Task(summary),              # Generate summary
    Task(lambda inputs: [translate(x, "fr") for x in inputs])  # Translate to French
])

# Step 3: Run the workflow
results = list(workflow(["https://neuml.com"]))

Sources: examples/workflow_quickstart.py:1-30

LLM-Powered Workflow Example

from txtai import LLM, Workflow
from txtai.pipeline import Textractor
from txtai.workflow import Task

textractor = Textractor(backend="docling")
llm = LLM("Qwen/Qwen3-4B-Instruct-2507")

def summary(text):
    return f"""Summarize the following text in 40 words or less.
{text}
"""

def translate(text, language):
    return f"""Translate the following text to {language}.
{text}
"""

workflow = Workflow([
    Task(textractor),
    Task(lambda inputs: llm([summary(x) for x in inputs], maxlength=25000)),
    Task(lambda inputs: llm([translate(x, "fr") for x in inputs], maxlength=25000))
])

results = list(workflow(["https://neuml.com"]))

Sources: examples/workflow_quickstart.py:35-60

Data Flow

graph LR
    A[(Input Data)] --> B[Workflow]
    B --> C[Task 1: Textractor]
    C --> D[Intermediate Data]
    D --> E[Task 2: Summary]
    E --> F[Intermediate Data]
    F --> G[Task N: Custom Action]
    G --> H[(Output Results)]
    
    I[URL Input] -->|UrlTask| C
    J[File Input] -->|FileTask| C
    K[Stream Input] -->|StreamTask| C

Input/Output Handling

• URL input: Processed via UrlTask wrapper
• File input: Processed via FileTask wrapper
• Stream input: Processed via StreamTask wrapper
• Tuple unpacking: Supports (id, data, tag) format for labeled data

Sources: src/python/txtai/workflow/base.py:60-100

Advanced Configuration

Custom Task Actions

from txtai.workflow import Task

def custom_filter(element):
    """Filter elements before processing"""
    return len(element) > 10

def custom_action(element):
    """Process single element"""
    return element.upper()

def custom_finalize(results):
    """Aggregate results after processing"""
    return sorted(results, key=lambda x: x[1], reverse=True)

workflow = Workflow([
    Task(
        action=custom_action,
        select=custom_filter,
        finalize=custom_finalize,
        concurrency="thread"
    )
])

Workflow Factory

from txtai.workflow import WorkflowFactory

# Create workflow from configuration
workflow = WorkflowFactory.create(config)

# Run with data
results = list(workflow(data))

Sources: src/python/txtai/workflow/factory.py:1-50

Best Practices

1. Use appropriate concurrency: Set concurrency="thread" for I/O-bound tasks (API calls, file operations) and concurrency="process" for CPU-intensive tasks.

1. Leverage one-to-many transforms: Tasks can produce multiple outputs from single inputs, enable with onetomany=True.

1. Template formatting: Use TemplateTask for consistent prompt formatting across LLM calls.

1. Error handling: Implement select filters to skip invalid data before reaching action processing.

1. Pipeline selection: Only install necessary pipeline extras—e.g., pip install txtai[pipeline-data] for text extraction workflows.

Sources: examples/workflow_quickstart.py:1-25

See Also

• Embeddings - Vector search and similarity indexing
• Pipelines - Language model pipelines
• LLM Configuration - Large language model setup
• API Documentation - REST API reference

Related Pages

Related topics: Pipelines, Workflows

Agents

Agents in txtai enable autonomous task execution by combining Large Language Models (LLMs) with a flexible tool system. The agent framework allows building AI-powered workflows that can reason, plan, and execute actions through configurable tools.

Overview

txtai agents are built on top of smolagents and provide:

• Tool-based execution: Agents use tools to interact with external systems and data sources
• LLM orchestration: Seamless integration with various LLM backends including local models, APIs, and managed services
• MCP support: Integration with Model Context Protocol for extended tool collections
• Configurable behavior: Flexible configuration for tool selection, prompt templates, and execution parameters

The agent architecture separates concerns between tool definition, tool management, and agent execution, enabling modular and extensible agent systems.

Architecture

graph TD
    A[User Input] --> B[Agent Core]
    B --> C[LLM Engine]
    C --> D[Tool Selection]
    D --> E[Tool Execution]
    E --> F[Results]
    F --> C
    C --> G[Final Response]
    
    H[ToolFactory] --> I[FunctionTool]
    H --> J[EmbeddingsTool]
    H --> K[MCPTools]
    H --> L[DefaultTools]
    
    subgraph Tool System
        I
        J
        K
        L
    end

Core Components

Tool System

Tool Types

The tool system supports multiple tool creation patterns:

FunctionTool

The FunctionTool class wraps descriptive configuration and injects it along with a target function into an LLM prompt.

from smolagents import Tool

class FunctionTool(Tool):
    """
    A FunctionTool takes descriptive configuration and injects it along with a target function into an LLM prompt.
    """

Configuration Schema:

Tool Factory

The ToolFactory class handles tool creation from various configuration formats:

@staticmethod
def create(config):
    """
    Creates a new list of tools. This method iterates of the `tools` configuration option and creates a Tool instance
    for each entry. This supports the following:

      - Tool instance
      - Dictionary with `name`, `description`, `inputs`, `output` and `target` function configuration
      - String with a tool alias name

    Returns:
        list of tools
    """

Supported Tool Input Formats:

1. Tool Instance: Direct smolagents.Tool objects
2. Dictionary: Config with name, description, inputs, output, and target
3. String Alias: Named shortcuts (e.g., "webview", "defaults")
4. MCP URL: HTTP URLs for MCP tool collections

Default Tools

Agent Configuration

Dependencies

The agent module requires the following dependencies (specified in setup.py:17):

extras["agent"] = [
    "jinja2>=3.1.6",
    "mcpadapt>=0.1.0",
    "smolagents>=1.23",
]

Configuration Options

Agent Execution Flow

sequenceDiagram
    participant U as User
    participant A as Agent
    participant L as LLM
    participant T as Tool Manager
    
    U->>A: Execute Task
    A->>L: Generate Tool Call
    L-->>A: Tool Selection
    A->>T: Execute Tool
    T-->>A: Tool Result
    A->>L: Process Result
    L-->>A: Response
    A-->>U: Final Answer

Integration with LLM Pipelines

Agents integrate with txtai's LLM pipeline system:

# From llm.py - Agent uses LLM generation
def generator(self, text, maxlength, stream, stop, defaultrole, stripthink, **kwargs):
    """
    Runs LLM generation for agent reasoning and tool calls.
    """
    return self.generator(text, maxlength, stream, stop, defaultrole, stripthink, **kwargs)

LLM Integration Methods

Tool Execution

The tool execution is handled by the forward method in FunctionTool:

def forward(self, *args, **kwargs):
    """
    Runs target function.

    Args:
        args: positional args
        kwargs: keyword args

    Returns:
        result
    """
    return self.target(*args, **kwargs)

Execution Parameters

Best Practices

1. Tool Descriptions: Provide clear, detailed descriptions for each tool to help the LLM understand when and how to use it
2. Input Schemas: Define complete input schemas with types and descriptions
3. Error Handling: Tools should handle errors gracefully and return meaningful error messages
4. Configuration Validation: Use the factory pattern to validate tool configurations before execution

See Also

• Pipeline LLM - LLM configuration and execution
• Workflows - Task orchestration
• Embeddings - Vector search capabilities

Related Pages

Related topics: Embeddings Database, Scoring and Retrieval Algorithms

Database Integration

Overview

txtai provides comprehensive database integration capabilities that enable vector search combined with structured data storage. The database layer is a core component of the embeddings database architecture, which unifies vector indexes (sparse and dense), graph networks, and relational databases into a single powerful system.

Sources: setup.py:42-44

The database integration serves as the foundation for:

• Persistent storage of indexed documents and embeddings
• SQL-based querying alongside vector similarity search
• Integration with external data sources and cloud storage
• Workflow task management and state persistence

Architecture

The database layer in txtai follows a modular architecture with several key components:

graph TD
    A[Embeddings Database] --> B[Database Layer]
    B --> C[SQLAlchemy Backend]
    B --> D[DuckDB Backend]
    C --> E[Schema Models]
    D --> E
    B --> F[Database Factory]
    F --> G[Database Client]
    G --> H[Query Engine]
    H --> I[Document Storage]
    H --> J[Vector Indexes]

Key Components

Supported Databases

SQLite/PostgreSQL with SQLAlchemy

txtai supports traditional relational databases through SQLAlchemy, providing cross-database compatibility. This enables seamless integration with:

• SQLite: Lightweight, file-based database for local development
• PostgreSQL: Production-grade relational database with advanced features
• pgvector: PostgreSQL extension for vector similarity search

Sources: setup.py:42-44

# Example configuration
embeddings:
    path: sentence-transformers/all-MiniLM-L6-v2
    database:
        url: postgresql+psycopg2://user:pass@localhost:5432/txtai

DuckDB

DuckDB is an embedded analytical database that provides high-performance OLAP capabilities. txtai includes DuckDB as a database extra for analytical workloads.

Sources: setup.py:42

extras["database"] = ["duckdb>=0.8.0", "pillow>=7.1.2", "sqlalchemy>=2.0.20"]

Database Configuration

Configuration Parameters

Connection String Formats

# SQLite (default)
sqlite:///txtai.db

# PostgreSQL with pgvector
postgresql://user:pass@localhost:5432/txtai

# DuckDB
duckdb:///path/to/database.duckdb

# MySQL
mysql+pymysql://user:pass@localhost:3306/txtai

Document Storage System

The document storage system provides efficient handling of indexed content with support for batch operations and streaming.

Core Operations

class DocumentStorage:
    def add(self, documents):
        """Add documents to storage"""
        
    def close(self):
        """Close and persist storage"""
        
    def delete(self, ids):
        """Remove documents by ID"""
        
    def get(self, ids):
        """Retrieve documents by ID"""

Sources: src/python/txtai/embeddings/index/documents.py:1-50

Batch Processing

Documents are processed in batches for optimal memory usage and performance:

# Batch-based document indexing
self.serializer.savestream(documents, self.documents)
self.batch += 1
self.size += len(documents)

Integration with Pipeline Data Processing

The Textractor module demonstrates database integration with external data sources, supporting extraction from various file formats and URLs.

Sources: src/python/txtai/pipeline/data/textractor.py:1-60

Supported Input Methods

Safe Open Mode

For security-sensitive environments, Textractor supports safe open mode that restricts access to local temporary directories and non-private URLs only:

self.safeopen = os.path.realpath(
    tempfile.gettempdir() if isinstance(safeopen, bool) else safeopen
) if safeopen else safeopen

Cloud Storage Integration

txtai extends database capabilities with cloud storage integration for managing large embeddings and document collections.

HuggingFace Hub Integration

The cloud hub module provides seamless upload and download of index files to HuggingFace repositories:

Sources: src/python/txtai/cloud/hub.py:1-60

# Automatic LFS tracking for large files
if "embeddings " not in content:
    content += "documents filter=lfs diff=lfs merge=lfs -text\n"
    content += "embeddings filter=lfs diff=lfs merge=lfs -text\n"

Supported Cloud Providers

Workflow Database Tasks

Workflows can include database operations as part of complex data processing pipelines.

Task Configuration

elif wtype == "tabular":
    data[wtype] = component
    tasks.append({"action": wtype})

Sources: examples/workflows.py:1-100

Workflow Task Types

Query Patterns

Vector + SQL Hybrid Queries

txtai supports combining vector similarity search with traditional SQL filtering:

# Semantic search with SQL filter
results = embeddings.search(
    "Find documents about machine learning",
    filter="category = 'technical' AND year > 2020"
)

Batch Query Optimization

For high-throughput applications:

# Batch query for multiple inputs
queries = ["query1", "query2", "query3"]
results = embeddings.batchsearch(queries, limit=10)

Performance Considerations

Database Selection Guide

Optimization Tips

1. Index Management: Regular index rebuilding for maintaining query performance
2. Batch Operations: Use batch operations for bulk inserts and updates
3. Connection Pooling: Configure appropriate pool sizes for concurrent access
4. Object Storage: Offload large embeddings to object storage while keeping metadata in database

Installation

Install database dependencies using extras:

# Basic database support
pip install txtai[database]

# With all database backends
pip install txtai[all]

# Individual backends
pip install txtai[duckdb]      # DuckDB analytics
pip install txtai[ann]         # Vector indexes with pgvector

See Also

• Embeddings Configuration - Detailed configuration guide
• Vector Search - Combining database with vector search
• Workflow Tasks - Database tasks in workflows
• Cloud Storage - HuggingFace Hub integration

Related Pages

Related topics: Embeddings Database, Database Integration

Graph Networks

Graph Networks in txtai provide a powerful mechanism for building and traversing graph-based knowledge structures. This module enables semantic relationship mapping, topic modeling, and community detection capabilities within the broader txtai ecosystem.

Overview

Graph Networks are a core component of txtai's architecture that combines vector search capabilities with graph-based relationship analysis. The graph module creates network representations that can be queried, traversed, and analyzed for patterns and communities.

graph TB
    subgraph "txtai Graph Module"
        GB[Graph Base Class]
        GX[NetworkX Backend]
        GF[Graph Factory]
        GT[Topics Module]
    end
    
    GB --> GX
    GB --> GT
    GF --> GB
    
    GX --> CD[Community Detection]
    GX --> NT[Node Traversal]
    GX --> RL[Relationship Links]

Sources: src/python/txtai/graph/base.py:1-20

Architecture

Base Graph Class

The Graph class serves as the foundational abstraction for all graph implementations. It provides a consistent interface for graph operations regardless of the underlying backend.

class Graph:
    """
    Base class for Graph instances. This class builds graph networks. Supports topic modeling
    and relationship traversal.
    """

Key Attributes:

Sources: src/python/txtai/graph/base.py:26-43

NetworkX Backend

The NetworkX backend (NetworkX class) provides the actual graph implementation using the NetworkX library. This backend supports:

• Community Detection: Uses Louvain algorithm for partition detection
• Graph Operations: Node/edge addition, removal, and querying
• Serialization: Pickle-based serialization for persistence
• Distance Computation: Weight-based distance calculations between nodes

Sources: src/python/txtai/graph/networkx.py:1-30

Core Methods

Graph Creation and Management

def create(self):
    """Creates the graph network."""
    raise NotImplementedError

def count(self):
    """Returns the total number of nodes in graph."""
    raise NotImplementedError

Sources: src/python/txtai/graph/base.py:45-65

Node Operations

Sources: src/python/txtai/graph/networkx.py:55-75

Community Detection

The NetworkX backend implements advanced community detection using the Louvain algorithm:

def communities(self, config):
    """
    Runs community detection on graph.

    Args:
        config: topic configuration

    Returns:
        list of [ids] per community
    """
    level = config.get("level", "best")
    results = list(louvain_partitions(self.backend, weight="weight", resolution=config.get("resolution", 100), seed=0))
    return results[0] if level == "first" else results[-1]

Configuration Parameters:

Sources: src/python/txtai/graph/networkx.py:35-55

Distance Computation

The graph module calculates distances between nodes based on edge weights:

def distance(self, source, target, attrs):
    """
    Computes distance between source and target nodes using weight.

    Returns:
        distance between source and target
    """
    distance = max(1.0 - attrs["weight"], 0.0)
    return distance if distance >= 0.15 else 1.00

Distance Algorithm:

• Distance = 1 - weight (inverted weight)
• Minimum threshold: 0.15 (edges below this are considered near-duplicates)
• Maximum distance cap: 1.00

Sources: src/python/txtai/graph/networkx.py:60-75

Serialization

TAR File Loading (Legacy Format)

The NetworkX backend supports loading graphs from legacy TAR archives:

def loadtar(self, path):
    """
    Loads a graph from the legacy TAR file.

    Args:
        path: path to graph
    """
    serializer = SerializeFactory.create("pickle")
    
    with TemporaryDirectory() as directory:
        archive = ArchiveFactory.create(directory)
        archive.load(path, "tar")
        self.backend = serializer.load(f"{directory}/graph")

Serialization Features:

• Pickle-based serialization for backwards compatibility
• TAR archive extraction to temporary directory
• Optional category loading from separate file

Sources: src/python/txtai/graph/networkx.py:80-100

Configuration

Graph networks are configured through a dictionary passed to the constructor:

graph:
    columns:
        text: "content"        # Column containing text data
        object: "data"        # Column containing object data
        relationships: "edges" # Column for manual edge definitions
    copyattributes: false      # Copy all attributes when True

Column Configuration Options:

Sources: docs/embeddings/configuration/graph.md and src/python/txtai/graph/base.py:38-40

Topic Modeling Integration

The Graph module integrates with txtai's topic modeling capabilities:

graph LR
    A[Input Data] --> B[Graph Indexing]
    B --> C[Topic Modeling]
    C --> D[Community Detection]
    D --> E[Category Assignment]
    
    C --> F[Topics Module]
    F --> G[Topic Analysis]

Topics Integration:

• self.categories: Stores discovered topic categories
• self.topics: Topics instance for analysis operations
• Community detection feeds into topic assignment

Sources: src/python/txtai/graph/base.py:30-32

Error Handling

The Graph module requires the NetworkX library for operation:

if not NETWORKX:
    raise ImportError('NetworkX is not available - install "graph" extra to enable')

To enable Graph Networks functionality, install the graph extra:

pip install txtai[graph]

Sources: src/python/txtai/graph/networkx.py:23-25

Usage Example

from txtai import Graph

# Create graph with default configuration
graph = Graph({})

# Create the graph network
graph.create()

# Add nodes with attributes
graph.addnode(1, text="example node", weight=0.5)
graph.addnode(2, text="another node", weight=0.7)

# Query nodes
nodes = list(graph.scan())
count = graph.count()

# Get community detection results
config = {"level": "best", "resolution": 100}
communities = graph.communities(config)

Related Components

Related Pages

Related topics: Embeddings Database, Database Integration

Scoring and Retrieval Algorithms

Overview

txtai provides a sophisticated scoring and retrieval system that powers its semantic search capabilities. At its core, the system combines vector-based similarity search with traditional BM25 sparse retrieval, enabling hybrid search approaches that leverage both semantic understanding and exact term matching.

The scoring module is an integral part of the embeddings database architecture, which unifies vector indexes, graph networks, and relational databases into a single powerful knowledge source for LLM applications.

Sources: README.md

Architecture

The scoring and retrieval system follows a modular factory pattern that allows different scoring algorithms to be plugged in based on configuration. The architecture is designed to support both standalone scoring operations and integrated retrieval within the embeddings database.

graph TD
    A[Query Input] --> B[Query Processing]
    B --> C{Scoring Algorithm}
    C --> D[BM25 Scoring]
    C --> E[Vector Similarity]
    C --> F[Hybrid Scoring]
    D --> G[Results Aggregation]
    E --> G
    F --> G
    G --> H[Ranked Results]

Sources: setup.py, src/python/txtai/scoring/factory.py

Scoring Module Structure

The scoring subsystem is defined in the txtai.scoring package and includes base classes and concrete implementations for different retrieval algorithms.

Key Components

Sources: setup.py:50-51

BM25 Retrieval Algorithm

BM25 (Best Matching 25) is a probabilistic ranking function used for information retrieval. It extends the binary independence model to incorporate term frequency and document length normalization.

Configuration Options

How BM25 Works

1. Term Frequency (TF): Measures how often a term appears in a document, with saturation to prevent over-weighting
2. Inverse Document Frequency (IDF): Weights terms by their rarity across the corpus
3. Document Length Normalization: Adjusts scores based on document length relative to average

Sources: src/python/txtai/scoring/bm25.py, src/python/txtai/scoring/base.py

Integration with Embeddings Database

The scoring algorithms are tightly integrated with the embeddings database system, which handles document indexing, storage, and retrieval.

graph LR
    A[Documents] --> B[Indexing Pipeline]
    B --> C[Vector Index]
    B --> D[BM25 Index]
    B --> E[Document Store]
    F[Query] --> G[Hybrid Search]
    C --> G
    D --> G
    E --> G
    G --> H[Scored Results]

Document Storage

Documents are managed through a streaming serializer that writes to temporary files, enabling efficient handling of large document collections:

# Add batch of documents
self.serializer.savestream(documents, self.documents)
self.batch += 1
self.size += len(documents)

Sources: src/python/txtai/embeddings/index/documents.py:18-23

Search API

The scoring system exposes search capabilities through both local and cluster APIs, supporting single queries, batch searches, and hybrid scoring with configurable weights.

Search Parameters

Query Execution Flow

sequenceDiagram
    participant Client
    participant API
    participant Scorer
    participant Index
    
    Client->>API: search(query, limit)
    API->>Scorer: execute(query)
    Scorer->>Index: score(query)
    Index-->>Scorer: raw scores
    Scorer-->>API: scored results
    API->>API: aggregate(query, results)
    API->>API: sort and limit
    API-->>Client: ranked results

Sources: src/python/txtai/api/cluster.py:85-105

Batch Search

For processing multiple queries efficiently, the API supports batch search operations:

def batchsearch(self, queries, limit=None, weights=None, index=None, parameters=None, graph=False):
    """
    Finds documents most similar to the input queries.
    """

Sources: src/python/txtai/api/cluster.py:107-127

Hybrid Scoring

txtai supports hybrid scoring that combines multiple retrieval methods. This is particularly valuable when you want to leverage both semantic similarity (vector-based) and exact term matching (BM25).

Weight Configuration

The weights parameter controls how different scoring methods are combined:

Result Aggregation

Results from multiple scoring methods are combined using weighted aggregation and sorted by the combined score:

# Combine aggregate functions and sort
results = self.aggregate(query, results)
# Limit results
return results[: (limit if limit else 10)]

Sources: src/python/txtai/api/cluster.py:97-99

Configuration Reference

Scoring Extra Dependencies

To use the scoring module, install the scoring extra:

pip install txtai[scoring]

Required dependencies from setup.py:

Sources: setup.py:50-51

Complete Installation Options

For full functionality including scoring:

pip install txtai[all]

This includes:

• ann - Approximate nearest neighbor indexes
• vectors - Vector similarity components
• scoring - Retrieval algorithms
• api - REST API service

Sources: setup.py:66-81

Usage Examples

Basic Embeddings Search

import txtai

embeddings = txtai.Embeddings()
embeddings.index(["Correct", "Not what we hoped"])
embeddings.search("positive", 1)
# Returns: [(0, 0.29862046241760254)]

Configuration via YAML

# app.yml
embeddings:
    path: sentence-transformers/all-MiniLM-L6-v2

Sources: README.md

Related Components

The scoring and retrieval algorithms form the computational backbone of txtai's search capabilities, enabling both high-precision keyword matching through BM25 and semantic understanding through vector similarity. This combination allows applications to benefit from the strengths of both traditional information retrieval and modern neural embeddings.

Doramagic Pitfall Log

Doramagic extracted 16 source-linked risk signals. Review them before installing or handing real data to the project.

1. Installation risk: Add `txtai_minimal` package

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add txtai_minimal package. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1090

2. Installation risk: Add custom Captions implementation

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add custom Captions implementation. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1084

3. Installation risk: Add custom Questions pipeline implementation

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add custom Questions pipeline implementation. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1087

4. Installation risk: Add custom Sequences pipeline implementation

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add custom Sequences pipeline implementation. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1086

5. Installation risk: Add custom Summary pipeline implementation

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add custom Summary pipeline implementation. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1085

6. Installation risk: Add minimal Docker build script

• Severity: medium
• Finding: Installation risk is backed by a source signal: Add minimal Docker build script. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1092

7. Installation risk: Make `transformers` package optional for minimal install

• Severity: medium
• Finding: Installation risk is backed by a source signal: Make transformers package optional for minimal install. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1091

8. Installation risk: Reduce dependencies to just `numpy` for minimal install

• Severity: medium
• Finding: Installation risk is backed by a source signal: Reduce dependencies to just numpy for minimal install. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1093

9. Installation risk: Support minimal install for edge devices

• Severity: medium
• Finding: Installation risk is backed by a source signal: Support minimal install for edge devices. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1089

10. Installation risk: Various training pipeline fixes for v5

• Severity: medium
• Finding: Installation risk is backed by a source signal: Various training pipeline fixes for v5. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1088

11. Installation risk: Zero dependency minimal install

• Severity: medium
• Finding: Installation risk is backed by a source signal: Zero dependency minimal install. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/issues/1094

12. Installation risk: v9.9.0

• Severity: medium
• Finding: Installation risk is backed by a source signal: v9.9.0. Treat it as a review item until the current version is checked.
• User impact: First-time setup may fail or require extra isolation and rollback planning.
• Recommended check: Open the linked source, confirm whether it still applies to the current version, and keep the first run isolated.
• Evidence: Source-linked evidence: https://github.com/neuml/txtai/releases/tag/v9.9.0

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using txtai with real data or production workflows.

• Add LiteRT-LM LLM - github / github_issue
• Zero dependency minimal install - github / github_issue
• Reduce dependencies to just numpy for minimal install - github / github_issue
• Make transformers package optional for minimal install - github / github_issue
• Add minimal Docker build script - github / github_issue
• Add txtai_minimal package - github / github_issue
• Various training pipeline fixes for v5 - github / github_issue
• Add custom Questions pipeline implementation - github / github_issue
• Add custom Sequences pipeline implementation - github / github_issue
• Add custom Summary pipeline implementation - github / github_issue
• Add custom Captions implementation - github / github_issue
• Support minimal install for edge devices - github / github_issue