# https://github.com/QuivrHQ/quivr Project Manual
Generated on: 2026-05-21 19:22:47 UTC
## Table of Contents
- [Introduction to Quivr](#page-introduction)
- [Getting Started](#page-getting-started)
- [Installation](#page-installation)
- [System Architecture Overview](#page-architecture-overview)
- [Core Components](#page-core-components)
- [Brain Class](#page-brain-class)
- [RAG Implementation](#page-rag-implementation)
- [LLM Integration](#page-llm-integration)
- [File Processing and Parsers](#page-file-processing)
- [Storage System](#page-storage)
## Introduction to Quivr
### Related Pages
Related topics: [Getting Started](#page-getting-started), [System Architecture Overview](#page-architecture-overview)
Relevant source files
The following source files were used to generate this page:
- [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
- [core/README.md](https://github.com/QuivrHQ/quivr/blob/main/core/README.md)
- [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
- [core/quivr_core/processor/implementations/simple_txt_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
- [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
- [core/CHANGELOG.md](https://github.com/QuivrHQ/quivr/blob/main/core/CHANGELOG.md)
# Introduction to Quivr
Quivr is an open-source project that helps you build your "second brain" by leveraging the power of Generative AI. It provides a Retrieval-Augmented Generation (RAG) framework that enables users to ingest documents and ask questions about their content using natural language. Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
The core philosophy of Quivr is to handle all the complexity of RAG implementations so developers can focus on their products. It provides an opinionated, fast, and efficient RAG system that supports multiple file types, various LLM providers, and customizable workflows. Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Key Features
Quivr offers several distinctive capabilities that make it a powerful choice for building document-aware AI applications:
| Feature | Description |
|---------|-------------|
| **Opinionated RAG** | Pre-configured RAG pipeline optimized for speed and efficiency |
| **Multi-LLM Support** | Works with OpenAI, Anthropic, Mistral, Gemma, and local models via Ollama |
| **Any File Type** | Supports PDF, TXT, Markdown, and custom parsers |
| **Customizable Workflows** | Extend RAG with internet search, tools, and custom processing |
| **Megaparse Integration** | Optional integration with Megaparse for advanced document ingestion |
| **Language Detection** | Automatic detection of document language for better processing |
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Architecture Overview
Quivr follows a modular architecture with the `quivr-core` package as its central component. The architecture is designed around a workflow-based system where different processing nodes are connected to form a complete RAG pipeline.
```mermaid
graph TD
A[User Input] --> B[Brain.ask]
B --> C[RetrievalConfig]
C --> D[Workflow Engine]
D --> E[Processing Nodes]
E --> F[LLM Response]
G[Documents] --> H[Processor]
H --> I[Chunks]
I --> J[Vector Store]
J --> D
```
### Package Structure
The main components of the Quivr architecture include:
| Component | Purpose |
|-----------|---------|
| `quivr_core.Brain` | Central class for managing document collections and answering questions |
| `ProcessorBase` | Abstract base class for document processors |
| `RetrievalConfig` | Configuration for retrieval and workflow settings |
| `QuivrFile` | File wrapper for document handling |
| `ProcessedDocument` | Container for processed document chunks |
Source: [core/README.md](https://github.com/QuivrHQ/quivr/blob/main/core/README.md)
## Getting Started
### Prerequisites
Before installing Quivr, ensure you have:
- Python 3.10 or newer
- An API key for your chosen LLM provider (OpenAI, Anthropic, or Mistral)
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Installation
Install the `quivr-core` package using pip:
```bash
pip install quivr-core
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Quick Start Example
The following example demonstrates creating a simple RAG application with Quivr in approximately 5 lines of code:
```python
import tempfile
from quivr_core import Brain
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt") as temp_file:
temp_file.write("Gold is a liquid of blue-like colour.")
temp_file.flush()
brain = Brain.from_files(
name="test_brain",
file_paths=[temp_file.name],
)
answer = brain.ask("what is gold? answer in french")
print("answer:", answer)
```
Source: [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
## Core Components
### Brain Class
The `Brain` class is the central interface for interacting with Quivr. It manages document ingestion, storage, and querying.
**Key Methods:**
| Method | Description |
|--------|-------------|
| `Brain.from_files()` | Create a brain from one or more files |
| `brain.ask()` | Ask a question about the ingested documents |
| `brain.print_info()` | Display information about the brain |
**Typical Workflow:**
```mermaid
graph LR
A[Create Brain] --> B[from_files]
B --> C[Process Documents]
C --> D[Store Chunks]
D --> E[Ask Questions]
E --> F[Get Answers]
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Document Processors
Processors handle the parsing and chunking of different file types. The `ProcessorBase` abstract class defines the interface that all processors must implement.
```python
class ProcessorBase(ABC):
@abstractmethod
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[R]:
raise NotImplementedError
```
Source: [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
**Processing Pipeline:**
During document processing, Quivr performs the following operations:
1. Parse the input file using the appropriate processor
2. Split documents into chunks based on `SplitterConfig`
3. Add metadata including chunk index, version info, and detected language
4. Sanitize content by removing null characters and encoding issues
Source: [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
### Simple Txt Processor
The `SimpleTxtProcessor` is a built-in processor for handling plain text files. It uses recursive character splitting to divide documents into manageable chunks:
```python
def recursive_character_splitter(
doc: Document, chunk_size: int, chunk_overlap: int
) -> list[Document]:
assert chunk_overlap < chunk_size, "chunk_overlap is greater than chunk_size"
if len(doc.page_content) <= chunk_size:
return [doc]
chunk = Document(page_content=doc.page_content[:chunk_size], metadata=doc.metadata)
remaining = Document(
page_content=doc.page_content[chunk_size - chunk_overlap :],
metadata=doc.metadata,
)
return [chunk] + recursive_character_splitter(remaining, chunk_size, chunk_overlap)
```
Source: [core/quivr_core/processor/implementations/simple_txt_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
## Configuration
### Environment Setup
Set your API keys as environment variables before creating a brain:
```python
import os
os.environ["OPENAI_API_KEY"] = "my_openai_apikey"
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Retrieval Configuration
The `RetrievalConfig` class allows customization of the RAG workflow. Configuration can be loaded from YAML files:
```python
from quivr_core.config import RetrievalConfig
config_file_name = "./basic_rag_workflow.yaml"
retrieval_config = RetrievalConfig.from_yaml(config_file_name)
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Workflow Configuration
Workflows are defined using YAML files that specify processing nodes and their connections:
```yaml
workflow_config:
name: "standard RAG"
nodes:
- name: "START"
edges: ["filter_history"]
- name: "filter_history"
edges: ["rewrite"]
- name: "rewrite"
edges: ["retrieve"]
```
**Workflow Node Types:**
| Node | Function |
|------|----------|
| START | Entry point for the workflow |
| filter_history | Filters conversation history |
| rewrite | Rewrites the query for better retrieval |
| retrieve | Fetches relevant document chunks |
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Advanced Usage
### Custom Workflow with Rich Console
For interactive applications, Quivr can be combined with the `rich` library for enhanced console output:
```python
from quivr_core import Brain
from quivr_core.config import RetrievalConfig
from rich.console import Console
from rich.panel import Panel
from rich.prompt import Prompt
brain = Brain.from_files(
name="my smart brain",
file_paths=["./my_first_doc.pdf", "./my_second_doc.txt"],
)
config_file_name = "./basic_rag_workflow.yaml"
retrieval_config = RetrievalConfig.from_yaml(config_file_name)
console = Console()
while True:
question = Prompt.ask("[bold cyan]Question[/bold cyan]")
if question.lower() == "exit":
break
answer = brain.ask(question, retrieval_config=retrieval_config)
console.print(f"[bold green]Answer[/bold green]: {answer.answer}")
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Supported LLM Providers
Quivr integrates with multiple LLM providers:
| Provider | Model Support | Configuration |
|----------|---------------|---------------|
| OpenAI | GPT-4, GPT-3.5 | `OPENAI_API_KEY` |
| Anthropic | Claude family | `ANTHROPIC_API_KEY` |
| Mistral | Mistral models | `MISTRAL_API_KEY` |
| Ollama | Local models | `OLLAMA_BASE_URL` |
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Version History
Quivr follows semantic versioning for the `quivr-core` package. Key releases include:
| Version | Date | Key Changes |
|---------|------|-------------|
| 0.0.27 | 2024-12-16 | Max context tokens enforcement, megaparse SDK integration |
| 0.0.19 | 2024-10-21 | Beginning of quivr-core development |
| 0.0.13 | 2024-08-01 | Added parsers and tox tests |
| 0.0.2 | 2024-07-09 | Initial quivr-core package release |
Source: [core/CHANGELOG.md](https://github.com/QuivrHQ/quivr/blob/main/core/CHANGELOG.md)
## Documentation and Community
Additional resources for learning and contributing to Quivr:
| Resource | Description |
|----------|-------------|
| [Official Documentation](https://core.quivr.com/) | Comprehensive guides and API reference |
| [GitHub Issues](https://github.com/quivrhq/quivr/issues) | Bug reports and feature requests |
| [Discord Community](https://discord.gg/HUpRgp2HG8) | Real-time support and discussions |
| [Good First Issues](https://github.com/quivrhq/quivr/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) | Beginner-friendly contribution opportunities |
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## License
Quivr is licensed under the Apache 2.0 License, making it freely available for commercial and personal use. Source: [core/README.md](https://github.com/QuivrHQ/quivr/blob/main/core/README.md)
---
## Getting Started
### Related Pages
Related topics: [Introduction to Quivr](#page-introduction), [Installation](#page-installation)
Relevant source files
The following source files were used to generate this page:
- [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
- [core/README.md](https://github.com/QuivrHQ/quivr/blob/main/core/README.md)
- [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
- [core/quivr_core/brain/brain.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
- [core/quivr_core/files/file.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
- [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
- [examples/pdf_parsing_tika.py](https://github.com/QuivrHQ/quivr/blob/main/examples/pdf_parsing_tika.py)
# Getting Started
## Overview
Quivr is an open-source RAG (Retrieval-Augmented Generation) framework that enables developers to build AI-powered "second brain" applications. The `quivr-core` package provides the core RAG functionality, allowing users to ingest documents and query them using large language models. Source: [README.md:1-15](https://github.com/QuivrHQ/quivr/blob/main/README.md)
The framework is designed to be **opinionated, fast, and efficient**, abstracting away the complexity of document processing, vector storage, and LLM integration so developers can focus on building their products. Source: [README.md:27-30](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Prerequisites
Before getting started with Quivr, ensure your environment meets the following requirements:
| Requirement | Version | Description |
|-------------|---------|-------------|
| Python | 3.10+ | The programming language runtime |
| pip | Latest | Package installer for Python |
| API Key | - | Required for cloud LLM providers (OpenAI, Anthropic, Mistral) |
Source: [README.md:44-50](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Supported LLM Providers
Quivr supports multiple LLM providers:
| Provider | API Type | Notes |
|----------|----------|-------|
| OpenAI | API Key | Set `OPENAI_API_KEY` environment variable |
| Anthropic | API Key | Supports Claude models |
| Mistral | API Key | Supports Mistral AI models |
| Ollama | Local | For running models locally |
Source: [README.md:58-62](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Installation
### Package Installation
Install the core package using pip:
```bash
pip install quivr-core
```
Source: [README.md:53-55](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Verify Installation
To verify the installation worked correctly, you can import the package:
```python
from quivr_core import Brain
print("Quivr-core installed successfully!")
```
## Core Concepts
### Brain
The `Brain` is the central entity in Quivr that manages document ingestion and querying. It encapsulates:
- **LLM Integration**: The language model used for generating answers
- **Embedder**: The embedding model for vectorizing documents
- **Vector Database**: Storage for document embeddings and retrieval
- **File Processors**: Components that parse various file formats
Source: [core/quivr_core/brain/brain.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
### QuivrFile
Files are represented as `QuivrFile` objects with the following attributes:
| Attribute | Type | Description |
|-----------|------|-------------|
| `id` | UUID | Unique identifier for the file |
| `brain_id` | UUID | ID of the brain this file belongs to |
| `path` | Path | File system path |
| `original_filename` | str | Original filename |
| `file_size` | int | File size in bytes |
| `file_extension` | FileExtension | File type enum |
| `file_sha1` | str | SHA1 hash for deduplication |
| `additional_metadata` | dict | Custom metadata |
Source: [core/quivr_core/files/file.py:50-70](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
### Processor Pipeline
Documents go through a processing pipeline that:
1. **Parses** the file content based on file type
2. **Splits** content into manageable chunks
3. **Detects language** for each chunk
4. **Embeds** chunks for vector storage
5. **Stores** in the vector database
Source: [core/quivr_core/processor/processor_base.py:40-60](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
## Quick Start Guide
### Minimal Example (5 Lines of Code)
The fastest way to get started with Quivr:
```python
import tempfile
from quivr_core import Brain
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt") as temp_file:
temp_file.write("Gold is a liquid of blue-like colour.")
temp_file.flush()
brain = Brain.from_files(
name="test_brain",
file_paths=[temp_file.name],
)
answer = brain.ask("what is gold? answer in french")
print("answer:", answer)
```
Source: [examples/simple_question/simple_question.py:1-20](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
### Interactive Chat Example
For a more complete example with a console-based chat interface:
```python
import os
os.environ["OPENAI_API_KEY"] = "your-api-key-here"
from quivr_core import Brain
from quivr_core.config import RetrievalConfig
brain = Brain.from_files(
name="my_smart_brain",
file_paths=["./document.pdf", "./notes.txt"],
)
answer = brain.ask(
"What is the main topic of these documents?",
retrieval_config=RetrievalConfig()
)
print(answer.answer)
```
Source: [README.md:85-100](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### PDF Processing Example
For processing PDF files with custom LLM configuration:
```python
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models import FakeListChatModel
from quivr_core import Brain
from quivr_core.rag.entities.config import LLMEndpointConfig
from quivr_core.llm.llm_endpoint import LLMEndpoint
brain = Brain.from_files(
name="test_brain",
file_paths=["tests/processor/data/dummy.pdf"],
llm=LLMEndpoint(
llm=FakeListChatModel(responses=["good"]),
llm_config=LLMEndpointConfig(model="fake_model", llm_base_url="local"),
),
embedder=DeterministicFakeEmbedding(size=20),
)
answer = brain.ask("What is this document about?")
print(answer.answer)
```
Source: [examples/pdf_parsing_tika.py:1-25](https://github.com/QuivrHQ/quivr/blob/main/examples/pdf_parsing_tika.py)
## Workflow Architecture
### Basic RAG Workflow
The following diagram illustrates the basic RAG workflow in Quivr:
```mermaid
graph TD
A[User Query] --> B[Filter History]
B --> C[Query Rewrite]
C --> D[Retrieval]
D --> E[LLM Generation]
E --> F[Response]
G[Document Ingestion] --> H[File Processing]
H --> I[Chunking]
I --> J[Embedding]
J --> K[Vector Storage]
D --> K
```
### Data Flow
```mermaid
graph LR
A[Files] -->|Parse| B[Documents]
B -->|Chunk| C[Chunks]
C -->|Embed| D[Vectors]
D -->|Store| E[Vector DB]
F[Query] -->|Embed| G[Query Vector]
G -->|Search| E
E -->|Results| H[Context]
H -->|Generate| I[Answer]
```
## Configuration
### Environment Variables
Configure your API keys as environment variables:
```bash
export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key" # Optional
export MISTRAL_API_KEY="your-mistral-key" # Optional
```
### Custom Retrieval Configuration
Create a YAML configuration file for customized retrieval strategies:
```yaml
workflow_config:
name: "standard RAG"
nodes:
- name: "START"
edges: ["filter_history"]
- name: "filter_history"
edges: ["rewrite"]
- name: "rewrite"
edges: ["retrieval"]
- name: "retrieval"
edges: ["generation"]
- name: "generation"
edges: ["END"]
llm:
temperature: 0.7
```
Source: [README.md:65-85](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### RetrievalConfig Usage
```python
from quivr_core.config import RetrievalConfig
# Load from YAML file
retrieval_config = RetrievalConfig.from_yaml("./basic_rag_workflow.yaml")
# Use with brain.ask()
answer = brain.ask(
question="Your question here",
retrieval_config=retrieval_config
)
```
## Supported File Types
Quivr works with various file formats through pluggable processors:
| File Type | Extension | Processor |
|-----------|-----------|-----------|
| Plain Text | `.txt` | SimpleTxtProcessor |
| PDF | `.pdf` | TikaProcessor |
| Markdown | `.md` | MarkdownProcessor |
| CSV | `.csv` | CSVProcessor |
| JSON | `.json` | JSONProcessor |
Source: [README.md:31-35](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Advanced Usage
### Streaming Responses
For real-time response streaming:
```python
from quivr_core import Brain
brain = Brain.from_files(
name="streaming_brain",
file_paths=["./documents/"],
)
for chunk in brain.answer_astream("Explain the findings"):
if chunk.last_chunk:
break
print(chunk.answer, end="", flush=True)
```
### Custom Processors
Implement your own processor by extending `ProcessorBase`:
```python
from quivr_core.processor.processor_base import ProcessorBase, ProcessedDocument
class CustomProcessor(ProcessorBase):
supported_extensions = [".custom"]
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument:
# Implement custom parsing logic
pass
```
## Project Structure
```
quivr/
├── README.md # Main documentation
├── core/
│ ├── README.md # quivr-core package info
│ └── quivr_core/
│ ├── brain/
│ │ └── brain.py # Brain class implementation
│ ├── files/
│ │ └── file.py # QuivrFile dataclass
│ ├── processor/
│ │ ├── processor_base.py
│ │ └── implementations/
│ └── rag/
│ ├── quivr_rag.py # RAG implementation
│ └── prompts.py # LLM prompts
├── examples/
│ ├── simple_question/ # Basic usage examples
│ ├── chatbot/ # Chainlit chatbot example
│ └── pdf_parsing_tika.py # PDF processing example
```
## Next Steps
After completing the Getting Started guide:
1. **Explore Examples**: Check the `examples/` directory for more use cases
2. **Read Documentation**: Visit [core.quivr.com](https://core.quivr.com/) for full documentation
3. **Join Community**: Connect with other users on [Discord](https://discord.gg/HUpRgp2HG8)
4. **Contribute**: Check [open issues](https://github.com/quivrhq/quivr/issues) for contribution opportunities
## Troubleshooting
### Common Issues
| Issue | Solution |
|-------|----------|
| ImportError: No module named 'quivr_core' | Run `pip install quivr-core` |
| API Key not found | Set environment variable before running |
| File parsing fails | Verify file format and size (max 20MB for examples) |
| Slow retrieval | Adjust `n_results` parameter or use local embeddings |
### Getting Help
- **Documentation**: [https://core.quivr.com/](https://core.quivr.com/)
- **Discord**: [Join our community](https://discord.gg/HUpRgp2HG8)
- **GitHub Issues**: [Report bugs](https://github.com/quivrhq/quivr/issues)
---
## Installation
### Related Pages
Related topics: [Getting Started](#page-getting-started), [LLM Integration](#page-llm-integration)
Relevant source files
The following source files were used to generate this page:
- [core/README.md](https://github.com/QuivrHQ/quivr/blob/main/core/README.md)
- [examples/chatbot/README.md](https://github.com/QuivrHQ/quivr/blob/main/examples/chatbot/README.md)
- [examples/chatbot_voice/README.md](https://github.com/QuivrHQ/quivr/blob/main/examples/chatbot_voice/README.md)
- [examples/quivr-whisper/README.md](https://github.com/QuivrHQ/quivr/blob/main/examples/quivr-whisper/README.md)
- [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
# Installation
This guide covers all aspects of installing and setting up Quivr, including the core package, examples, and required dependencies.
## Overview
Quivr is a RAG (Retrieval-Augmented Generation) framework that enables users to create AI-powered knowledge bases from various file types. The installation process varies depending on your use case:
- **Core package installation** for integrating Quivr into existing Python projects
- **Example applications** for testing and learning purposes
- **Development setup** for contributing to the project
Source: [core/README.md]()
## Prerequisites
### System Requirements
| Requirement | Minimum | Recommended |
|-------------|---------|-------------|
| Python Version | 3.8+ | 3.10+ |
| Operating System | Linux, macOS, Windows | Linux/macOS |
| RAM | 4 GB | 8 GB+ |
| Disk Space | 500 MB | 1 GB+ |
**Note:** While the core package officially requires Python 3.10 or newer for full compatibility, some examples (such as chatbot_voice) support Python 3.8 or higher.
Source: [core/README.md](), [examples/chatbot_voice/README.md]()
### API Keys
Quivr supports multiple LLM providers. You must configure at least one API key:
| Provider | Environment Variable | Required |
|----------|---------------------|----------|
| OpenAI | `OPENAI_API_KEY` | Yes (if using OpenAI) |
| Anthropic | `ANTHROPIC_API_KEY` | Yes (if using Anthropic) |
| Mistral | `MISTRAL_API_KEY` | Yes (if using Mistral) |
Source: [core/README.md]()
## Installing the Core Package
### Using pip (Recommended)
The simplest way to install Quivr Core is via pip:
```bash
pip install quivr-core
```
Verify the installation by checking that the package is importable:
```python
from quivr_core import Brain
```
Source: [core/README.md]()
### Package Contents
The `quivr-core` package includes:
- **Brain class**: The main interface for creating and managing knowledge bases
- **RAG components**: Retrieval, processing, and answer generation pipelines
- **Built-in processors**: Support for PDF, TXT, Markdown, and other common formats
- **Default configurations**: Ready-to-use settings for quick setup
Source: [core/README.md](), [core/quivr_core/processor/processor_base.py]()
## Installation for Examples
### Chatbot Example with Chainlit
The chatbot example demonstrates file upload and Q&A capabilities using Chainlit.
#### Using rye (Recommended)
```bash
# Clone or navigate to the chatbot directory
cd examples/chatbot
# Install dependencies with rye
rye sync
# Activate the virtual environment
source ./venv/bin/activate
```
#### Using pip
```bash
# Navigate to the chatbot directory
cd examples/chatbot
# Install from requirements
pip install -r requirements.txt
```
Source: [examples/chatbot/README.md]()
### Voice Chatbot Example
The voice chatbot example adds voice interaction capabilities.
```bash
# Navigate to the voice chatbot directory
cd examples/chatbot_voice
# Install dependencies
pip install -r requirements.lock
```
Source: [examples/chatbot_voice/README.md]()
### Flask-based Example (quivr-whisper)
This example uses Flask for a web server implementation.
```bash
# Install Flask and dependencies
pip install flask openai requests python-dotenv
```
Source: [examples/quivr-whisper/README.md]()
### Simple Question Example
The simplest example demonstrating basic usage:
```bash
# Create a Python script with the following content
import tempfile
from quivr_core import Brain
brain = Brain.from_files(
name="test_brain",
file_paths=["your_file.txt"],
)
answer = brain.ask("Your question here")
print(answer)
```
Requires `python-dotenv` for loading environment variables.
Source: [examples/simple_question/simple_question.py]()
## Environment Configuration
### Required Environment Variables
Create a `.env` file in your project root:
```env
# LLM API Keys (at least one required)
OPENAI_API_KEY=your_openai_api_key
# ANTHROPIC_API_KEY=your_anthropic_key
# MISTRAL_API_KEY=your_mistral_key
# Optional: For specific integrations
QUIVR_API_KEY=your_quivr_api_key
QUIVR_CHAT_ID=your_chat_id
QUIVR_BRAIN_ID=your_brain_id
QUIVR_URL=https://api.quivr.app
```
### Loading Environment Variables
Use `python-dotenv` to load environment variables:
```python
import dotenv
dotenv.load_dotenv()
```
Source: [examples/quivr-whisper/README.md](), [examples/simple_question/simple_question.py]()
## Installation Flow
```mermaid
graph TD
A[Start Installation] --> B{Choose Installation Type}
B --> C[Core Package Only]
B --> D[Examples & Demos]
B --> E[Development Setup]
C --> F[pip install quivr-core]
F --> G[Set API Keys]
G --> H[Ready to Integrate]
D --> I{Which Example?}
I --> J[Chatbot with Chainlit]
I --> K[Voice Chatbot]
I --> L[Flask App]
J --> M[rye sync or pip install]
K --> N[pip install -r requirements.lock]
L --> O[pip install flask openai requests]
M --> P[Run with chainlit run main.py]
N --> Q[Run with chainlit run main.py]
O --> R[Run with flask run]
E --> S[Clone Repository]
S --> T[Navigate to core/]
T --> U[Install from pyproject.toml]
U --> V[Run Tests]
```
## Verifying Installation
### Quick Verification
After installation, verify that Quivr is correctly installed:
```python
import quivr_core
print(quivr_core.__version__) # Should print the installed version
```
### Full Installation Test
Create a test script to verify all components:
```python
import tempfile
from quivr_core import Brain
# Create a temporary test file
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f:
f.write("Quivr is a RAG framework for building AI knowledge bases.")
temp_path = f.name
# Create a brain from the file
brain = Brain.from_files(
name="test_brain",
file_paths=[temp_path],
)
# Test the ask function
answer = brain.ask("What is Quivr?")
print(f"Answer: {answer}")
# Print brain info
brain.print_info()
```
Source: [core/README.md](), [examples/simple_question/simple_question.py]()
## Troubleshooting
### Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| ImportError: No module named 'quivr_core' | Package not installed | Run `pip install quivr-core` |
| AuthenticationError | Invalid API key | Verify your API key is correct |
| Version mismatch | Incompatible Python version | Ensure Python 3.10+ is used |
| File not found | Incorrect file path | Check the file path exists |
### Checking Installed Version
The Quivr version is automatically tracked in document metadata:
```python
from quivr_core.processor.processor_base import get_version
try:
from importlib.metadata import version
qvr_version = version("quivr-core")
except PackageNotFoundError:
qvr_version = "dev"
```
Source: [core/quivr_core/processor/processor_base.py]()
## Next Steps
After successful installation:
1. **Create your first Brain**: Load documents and create a knowledge base
2. **Configure RAG**: Customize retrieval strategies using YAML configuration files
3. **Explore Examples**: Test different example applications to understand capabilities
4. **Read Documentation**: Visit [core.quivr.com](https://core.quivr.com/) for advanced usage
Source: [core/README.md](), [examples/chatbot/README.md]()
---
## System Architecture Overview
### Related Pages
Related topics: [Core Components](#page-core-components), [Brain Class](#page-brain-class), [RAG Implementation](#page-rag-implementation)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/brain/brain.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
- [core/quivr_core/rag/quivr_rag.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
- [core/quivr_core/rag/quivr_rag_langgraph.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag_langgraph.py)
- [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
- [core/quivr_core/files/file.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
- [core/quivr_core/rag/prompts.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/prompts.py)
# System Architecture Overview
## Introduction
Quivr is an open-source framework for building RAG (Retrieval Augmented Generation) applications that enables users to create personal "brains" from various document types. The system leverages generative AI to provide intelligent question-answering capabilities over user-provided documents.
The architecture is designed around three core pillars:
- **Document Processing**: Extracting and chunking content from files
- **Vector Storage**: Storing embeddings for semantic search
- **LLM-powered Answer Generation**: Using large language models to generate answers based on retrieved context
## High-Level Architecture
```mermaid
graph TD
A[User Files] --> B[File Processing]
B --> C[Chunking & Embedding]
C --> D[Vector Database]
E[User Query] --> F[Embedding]
F --> G[Semantic Search]
G --> H[Context Assembly]
H --> I[LLM Generation]
I --> J[Answer Response]
D --> G
```
## Core Components
### Brain Class
The `Brain` class is the central orchestrator of the Quivr framework. It manages the entire lifecycle from file ingestion to query answering.
**File**: `core/quivr_core/brain/brain.py`
#### Key Responsibilities
| Responsibility | Description |
|----------------|-------------|
| File Ingestion | Processes files through various parsers |
| Vector Storage | Manages the vector database for embeddings |
| Query Processing | Handles search and retrieval operations |
| Answer Generation | Orchestrates LLM-based response generation |
#### Core Methods
| Method | Type | Purpose |
|--------|------|---------|
| `from_files` | Synchronous | Create a Brain from file paths |
| `afrom_files` | Async | Async creation from files |
| `afrom_langchain_documents` | Async | Create from LangChain Document objects |
| `asearch` | Async | Search for relevant documents |
| `ask_streaming` | Async Generator | Stream answers to questions |
#### Initialization Parameters
```python
class Brain:
def __init__(
self,
id: UUID,
name: str,
storage: BrainStorage | None,
llm: LLMEndpoint,
embedder: Embeddings,
vector_db: QuivrVectorStore,
)
```
Source: `core/quivr_core/brain/brain.py:1-100`
### File Processing Pipeline
The processor system handles extraction and transformation of various file formats.
**File**: `core/quivr_core/processor/processor_base.py`
#### Processing Flow
```mermaid
graph LR
A[QuivrFile] --> B[process_file]
B --> C[process_file_inner]
C --> D[ProcessedDocument]
D --> E[Metadata Enrichment]
E --> F[Chunk Output]
```
#### Metadata Enrichment
During processing, each chunk receives comprehensive metadata:
```python
doc.metadata = {
"chunk_index": idx,
"quivr_core_version": qvr_version,
"language": detect_language(text=...),
**file.metadata,
**doc.metadata,
**self.processor_metadata,
}
```
Source: `core/quivr_core/processor/processor_base.py:20-35`
### QuivrFile Data Model
**File**: `core/quivr_core/files/file.py`
The `QuivrFile` class represents uploaded files with their associated metadata.
```python
class QuivrFile:
__slots__ = [
"id",
"brain_id",
"path",
"original_filename",
"file_size",
"file_extension",
"file_sha1",
"additional_metadata",
]
```
| Field | Type | Description |
|-------|------|-------------|
| `id` | UUID | Unique identifier |
| `brain_id` | UUID | Associated brain identifier |
| `path` | Path | File system path |
| `original_filename` | str | Original file name |
| `file_size` | int | File size in bytes |
| `file_extension` | FileExtension | File type |
| `file_sha1` | str | SHA1 hash for deduplication |
Source: `core/quivr_core/files/file.py:1-50`
### Retrieval and Answer Generation
## RAG Architecture
The RAG (Retrieval Augmented Generation) system combines semantic search with LLM-powered answer generation.
**File**: `core/quivr_core/rag/quivr_rag_langgraph.py`
#### RAG Workflow
```mermaid
graph TD
A[User Question] --> B[Filter History]
B --> C[Query Rewrite]
C --> D[Retrieval]
D --> E[Context Assembly]
E --> F[LLM Generation]
F --> G[Streaming Response]
H[System Prompt] --> F
I[Chat History] --> B
J[File Filters] --> D
```
#### Configuration
The system uses `RetrievalConfig` for configuring the RAG pipeline:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `llm_config` | LLMEndpointConfig | Brain's LLM | LLM model configuration |
| `temperature` | float | 0.7 | Generation temperature |
| `n_results` | int | 5 | Number of retrieval results |
Source: `core/quivr_core/rag/quivr_rag.py:1-50`
### Streaming Answer Generation
The system supports streaming responses for real-time answer delivery:
```python
async for response in rag_instance.answer_astream(
run_id=run_id,
question=question,
system_prompt=system_prompt or None,
history=chat_history,
list_files=list_files,
metadata=metadata,
):
if not response.last_chunk:
yield response
```
Source: `core/quivr_core/brain/brain.py:150-170`
## Prompt Templates
**File**: `core/quivr_core/rag/prompts.py`
The system uses structured prompts with multiple context sections:
| Section | Purpose |
|---------|---------|
| `user_metadata` | User-specific context |
| `ticket_metadata` | Query-related metadata |
| `similar_tickets` | Reference information |
| `ticket_history` | Conversation history |
| `additional_information` | Extra context |
| `client_query` | The actual question |
Default instructions ensure consistent response quality:
- Verbosity matching similar responses
- Proper formatting with paragraphs, bold, italic
- Language consistency with the query
- No signature at end of response
## LLM Integration
### Default LLM Configuration
The system supports multiple LLM providers:
| Provider | Configuration |
|----------|---------------|
| OpenAI | `OPENAI_API_KEY` environment variable |
| Anthropic | Anthropic API support |
| Mistral | Mistral API support |
| Ollama | Local model support |
Source: README.md - Configuration section
### Embedder Abstraction
Embedders are abstracted to support multiple backends:
```python
if embedder is None:
embedder = default_embedder()
```
Vector DB initialization with embeddings:
```python
if vector_db is None:
vector_db = await build_default_vectordb(langchain_documents, embedder)
```
Source: `core/quivr_core/brain/brain.py:60-70`
## Search Capabilities
### Async Search Method
```python
async def asearch(
self,
query: str | Document,
n_results: int = 5,
filter: Callable | Dict[str, Any] | None = None,
fetch_n_neighbors: int = 20,
) -> list[SearchResult]
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | str \| Document | Required | Search query |
| `n_results` | int | 5 | Number of results |
| `filter` | Callable \| Dict | None | Custom filtering |
| `fetch_n_neighbors` | int | 20 | Extended fetch for re-ranking |
Source: `core/quivr_core/brain/brain.py:70-85`
## State Management
### Brain ID Generation
Each brain receives a unique identifier:
```python
brain_id = uuid4()
```
### Workspace and Chat Context
The system tracks conversation context:
```python
metadata = LangchainMetadata(
langfuse_trace_id=str(run_id),
langfuse_user_id=str(self.workspace_id),
langfuse_session_id=str(self.chat_id),
)
```
## Installation and Dependencies
### Core Package
```bash
pip install quivr-core
```
### Quick Start Example
```python
from quivr_core import Brain
brain = Brain.from_files(
name="my_smart_brain",
file_paths=["./my_first_doc.pdf", "./my_second_doc.txt"],
)
answer = brain.ask("What is the main topic?")
```
## Summary
The Quivr architecture implements a modular RAG pipeline where:
1. **Files** are processed and chunked with metadata enrichment
2. **Embeddings** are generated and stored in a vector database
3. **Queries** trigger semantic search with configurable filters
4. **LLMs** generate contextual answers from retrieved content
5. **Streaming** enables real-time response delivery
The design prioritizes flexibility through dependency injection, allowing custom LLM providers, embedders, and vector stores while maintaining a consistent API for brain creation and querying.
---
## Core Components
### Related Pages
Related topics: [System Architecture Overview](#page-architecture-overview), [Brain Class](#page-brain-class), [LLM Integration](#page-llm-integration)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/brain/brain.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
- [core/quivr_core/rag/quivr_rag.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
- [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
- [core/quivr_core/llm/llm_endpoint.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/llm/llm_endpoint.py)
- [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
- [core/quivr_core/processor/implementations/simple_txt_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
- [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
# Core Components
Quivr is an open-source RAG (Retrieval-Augmented Generation) framework that enables users to create "brains" from various document types and query them using natural language. The core components form the architectural foundation that powers document processing, vector storage, retrieval, and LLM-based answer generation.
## Architecture Overview
Quivr's architecture follows a modular design with clear separation of concerns across several key layers:
```mermaid
graph TD
A[User Query] --> B[Brain.ask]
B --> C[RetrievalConfig]
C --> D[QuivrQARAGLangGraph]
D --> E[Vector Store Retrieval]
E --> F[Context Chunks]
F --> G[LLM Generation]
G --> H[Streaming Response]
I[Files] --> J[Processor]
J --> K[ProcessedDocument]
K --> L[Vector DB Indexing]
L --> E
M[LLM Config] --> G
N[Embedder] --> L
```
The system is built around three primary abstractions:
| Component | Purpose | Key Files |
|-----------|---------|-----------|
| **Brain** | Central orchestrator managing files, vectors, and LLM interactions | brain.py |
| **RAG Engine** | Handles retrieval and answer generation pipeline | quivr_rag.py |
| **Processors** | Parse and chunk various file formats | processor_base.py |
---
## Brain Class
The `Brain` class is the central entry point for all Quivr operations. It encapsulates file management, vector storage, and LLM-based question answering.
### Initialization Methods
The Brain class provides multiple factory methods for creation:
| Method | Description | Use Case |
|--------|-------------|----------|
| `from_files()` | Synchronous creation from file paths | Simple scripts, examples |
| `afrom_files()` | Async creation from file paths | Production async applications |
| `from_langchain_documents()` | From pre-processed LangChain documents | Advanced integration scenarios |
| `afrom_langchain_documents()` | Async version | Async workflows with external processing |
### Creating a Brain from Files
```python
from quivr_core import Brain
brain = Brain.from_files(
name="my_smart_brain",
file_paths=["./my_first_doc.pdf", "./my_second_doc.txt"],
)
```
The `from_files()` method automatically:
1. Processes each file using the appropriate processor based on file extension
2. Chunks documents according to the configured splitter settings
3. Generates embeddings using the configured embedder
4. Stores vectors in the configured vector database
5. Returns an initialized Brain instance ready for queries
Source: [examples/simple_question/simple_question.py:1-17](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
### Querying the Brain
```python
answer = brain.ask(
"what is gold? answer in french",
retrieval_config=retrieval_config
)
print("answer:", answer.answer)
```
The `ask()` method supports:
- Streaming responses via `ask_streaming()`
- Custom retrieval configurations
- Chat history management
- System prompt customization
- File filtering during retrieval
Source: [core/quivr_core/brain/brain.py:150-200](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
---
## Retrieval Configuration
The `RetrievalConfig` class controls how documents are retrieved and how the LLM generates responses.
### Configuration Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `llm_config` | `LLMEndpointConfig` | Brain's default LLM | LLM model and endpoint settings |
| `n_results` | `int` | `5` | Number of documents to retrieve |
| `fetch_n_neighbors` | `int` | `20` | Neighbors to fetch for re-ranking |
| `llm_base_url` | `str` | `None` | Custom LLM endpoint base URL |
| `llm_api_key` | `str` | Environment var | API key for LLM service |
### YAML Configuration
Quivr supports YAML-based workflow configuration:
```yaml
workflow_config:
name: "standard RAG"
nodes:
- name: "START"
edges: ["filter_history"]
- name: "filter_history"
edges: ["rewrite"]
- name: "rewrite"
edges: ["retrieval"]
```
Configuration can be loaded from YAML files:
```python
from quivr_core.config import RetrievalConfig
retrieval_config = RetrievalConfig.from_yaml("./basic_rag_workflow.yaml")
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
---
## LLM Integration
The LLM layer is abstracted through the `LLMEndpoint` class, providing flexibility to use different LLM providers.
### Supported Providers
Quivr supports multiple LLM providers:
- **OpenAI** (GPT models)
- **Anthropic** (Claude models)
- **Mistral**
- **Local models** via Ollama
### LLM Configuration
```python
from quivr_core.llm.llm_endpoint import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
llm = LLMEndpoint.from_config(
config=LLMEndpointConfig(
model="gpt-4",
llm_base_url="https://api.openai.com/v1",
temperature=0.7
)
)
```
### Embedding Configuration
Embeddings are generated using the configured embedder. If no embedder is specified, Quivr uses the default embedder:
```python
if embedder is None:
embedder = default_embedder()
```
The embedder is critical for:
- Converting document chunks into vector representations
- Encoding user queries for similarity search
- Enabling semantic retrieval over the document corpus
Source: [core/quivr_core/llm/llm_endpoint.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/llm/llm_endpoint.py)
---
## File Processing Pipeline
The processor system handles parsing various file formats and converting them into chunked documents suitable for vector storage.
### Processor Architecture
```mermaid
graph LR
A[QuivrFile] --> B[ProcessorBase]
B --> C[process_file_inner]
C --> D[ProcessedDocument]
D --> E[Splitter]
E --> F[Chunked Documents]
F --> G[Metadata Enrichment]
G --> H[Vector DB Indexing]
```
### Processor Base Class
All processors inherit from `ProcessorBase`, which provides:
- Async file processing interface
- Automatic metadata enrichment
- Language detection
- Chunk indexing
```python
class ProcessorBase(ABC, Generic[R]):
@abstractmethod
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[R]:
raise NotImplementedError
```
### Metadata Enrichment
During processing, each chunk receives enriched metadata:
```python
doc.metadata = {
"chunk_index": idx,
"quivr_core_version": qvr_version,
"language": detect_language(
text=doc.page_content,
low_memory=True,
).value,
**file.metadata,
**doc.metadata,
**self.processor_metadata,
}
```
Source: [core/quivr_core/processor/processor_base.py:40-55](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
### Supported Processors
| Processor | Extensions | Description |
|-----------|------------|-------------|
| `SimpleTxtProcessor` | `.txt` | Plain text file processing |
| `PdfProcessor` | `.pdf` | PDF document parsing |
| `MarkdownProcessor` | `.md` | Markdown file parsing |
### Text File Processing
The `SimpleTxtProcessor` demonstrates the recursive chunking strategy:
```python
def recursive_character_splitter(
doc: Document, chunk_size: int, chunk_overlap: int
) -> list[Document]:
if len(doc.page_content) <= chunk_size:
return [doc]
chunk = Document(page_content=doc.page_content[:chunk_size], metadata=doc.metadata)
remaining = Document(
page_content=doc.page_content[chunk_size - chunk_overlap:],
metadata=doc.metadata,
)
return [chunk] + recursive_character_splitter(remaining, chunk_size, chunk_overlap)
```
Key constraints enforced:
- `chunk_overlap` must be less than `chunk_size`
- Original file names are prepended to chunk content
- Null characters are removed
- UTF-8 encoding is enforced
Source: [core/quivr_core/processor/implementations/simple_txt_processor.py:15-30](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
---
## RAG Engine
The RAG engine coordinates the retrieval and generation pipeline, transforming user queries into contextual answers.
### Answer Generation Flow
```mermaid
sequenceDiagram
participant User
participant Brain
participant RAG as QuivrQARAGLangGraph
participant VectorDB
participant LLM
User->>Brain: ask(question)
Brain->>RAG: answer_astream()
RAG->>VectorDB: search(query, n_results)
VectorDB-->>RAG: relevant_chunks
RAG->>LLM: generate(context, question)
LLM-->>RAG: streaming_response
RAG-->>User: yield chunks
```
### Streaming Responses
The `answer_streaming()` method returns an async generator:
```python
async for chunk in brain.ask_streaming("What is the meaning of life?"):
print(chunk.answer)
```
Each chunk contains:
- `answer`: The partial or complete response text
- `sources`: Relevant document metadata
- `last_chunk`: Boolean flag indicating completion
- `metadata`: Additional context about the generation
### Chat History Integration
The RAG engine maintains chat history for conversational queries:
```python
chat_history = self.default_chat if chat_history is None else chat_history
full_answer = ""
async for response in rag_instance.answer_astream(
question=question,
history=chat_history,
list_files=list_files,
metadata=metadata,
):
# Process streaming chunks
```
Source: [core/quivr_core/rag/quivr_rag.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
---
## Brain Serialization and Persistence
Brains can be serialized to disk and reloaded for persistence across sessions.
### Save Operation
```python
save_path = await brain.save("/home/user/.local/quivr")
```
### Load Operation
```python
brain_loaded = Brain.load(save_path)
brain_loaded.print_info()
```
Source: [examples/save_load_brain.py](https://github.com/QuivrHQ/quivr/blob/main/examples/save_load_brain.py)
---
## Search Functionality
The Brain class provides direct search capabilities for retrieving relevant documents without generating LLM responses.
### Search Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `query` | `str \| Document` | Search query text or LangChain document |
| `n_results` | `int` | Number of results to return (default: 5) |
| `filter` | `Callable \| Dict \| None` | Optional metadata filtering |
| `fetch_n_neighbors` | `int` | Neighbors to fetch for re-ranking (default: 20) |
### Search Return Type
The `asearch()` method returns a list of `SearchResult` objects containing:
- Matched document chunks
- Similarity scores
- Metadata about the source documents
---
## Configuration Summary
### Environment Variables
| Variable | Required | Description |
|----------|----------|-------------|
| `OPENAI_API_KEY` | Yes (for OpenAI) | API key for OpenAI models |
| `ANTHROPIC_API_KEY` | Yes (for Claude) | API key for Anthropic models |
| `OPENAI_API_BASE` | No | Custom API base URL |
### Default Configuration
When no configuration is provided, Quivr uses sensible defaults:
```python
if llm is None:
llm = default_llm()
if embedder is None:
embedder = default_embedder()
if vector_db is None:
vector_db = await build_default_vectordb(langchain_documents, embedder)
```
This ensures that users can get started with minimal configuration while still allowing full customization.
---
## Next Steps
- **Workflows**: Explore YAML-based workflow configurations for advanced retrieval strategies
- **Custom Processors**: Implement the `ProcessorBase` interface to support additional file formats
- **LLM Providers**: Configure different LLM providers based on your requirements
- **Deployment**: Review deployment documentation for production setups
---
## Brain Class
### Related Pages
Related topics: [RAG Implementation](#page-rag-implementation), [File Processing and Parsers](#page-file-processing), [Storage System](#page-storage)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/brain/brain.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
- [core/quivr_core/brain/__init__.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/__init__.py)
- [examples/save_load_brain.py](https://github.com/QuivrHQ/quivr/blob/main/examples/save_load_brain.py)
- [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
- [examples/pdf_parsing_tika.py](https://github.com/QuivrHQ/quivr/blob/main/examples/pdf_parsing_tika.py)
- [core/quivr_core/files/file.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
# Brain Class
The `Brain` class is the central abstraction in Quivr for building Retrieval Augmented Generation (RAG) systems. It serves as the primary interface for creating knowledge bases from documents, performing semantic search, and generating AI-powered answers based on retrieved context.
## Overview
The Brain class encapsulates:
- **Vector storage** for semantic document indexing
- **LLM integration** for answer generation
- **Embedder configuration** for document vectorization
- **File processing pipeline** for document ingestion
- **Chat history management** for conversational context
A Brain can be created from files (PDF, TXT, Markdown, etc.) or from LangChain documents, then queried to generate context-aware responses.
## Architecture
```mermaid
graph TD
A[Files / Documents] --> B[Brain Class]
B --> C[Vector DB]
B --> D[LLM]
B --> E[Embedder]
B --> F[Storage]
G[Query] --> B
B --> H[QuivrQARAGLangGraph]
H --> I[Answer]
C --> H
```
## Creating a Brain
### From Files
The most common way to create a Brain is from a collection of files:
```python
from quivr_core import Brain
brain = Brain.from_files(
name="my_smart_brain",
file_paths=["./my_first_doc.pdf", "./my_second_doc.txt"],
)
```
Source: [examples/simple_question/simple_question.py:1-14](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
### From LangChain Documents
For programmatic document creation:
```python
from langchain_core.documents import Document
from quivr_core import Brain
documents = [Document(page_content="Hello, world!")]
brain = await Brain.afrom_langchain_documents(name="My Brain", langchain_documents=documents)
```
Source: [core/quivr_core/brain/brain.py:1-150](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
### With Custom LLM and Embedder
Override default configurations with custom implementations:
```python
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models import FakeListChatModel
from quivr_core import Brain
from quivr_core.rag.entities.config import LLMEndpointConfig
from quivr_core.llm.llm_endpoint import LLMEndpoint
brain = Brain.from_files(
name="test_brain",
file_paths=["tests/processor/data/dummy.pdf"],
llm=LLMEndpoint(
llm=FakeListChatModel(responses=["good"]),
llm_config=LLMEndpointConfig(model="fake_model", llm_base_url="local"),
),
embedder=DeterministicFakeEmbedding(size=20),
)
```
Source: [examples/pdf_parsing_tika.py:1-20](https://github.com/QuivrHQ/quivr/blob/main/examples/pdf_parsing_tika.py)
## Core Methods
### Asking Questions
#### Synchronous
```python
answer = brain.ask(
"what is gold? answer in french",
retrieval_config=retrieval_config
)
print("answer:", answer)
```
#### Asynchronous Streaming
```python
async for chunk in brain.ask_streaming("What is the meaning of life?"):
print(chunk.answer)
```
Source: [core/quivr_core/brain/brain.py:150-250](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
### Search
#### Async Search
```python
results = await brain.asearch(
query="your search query",
n_results=5,
filter=None,
fetch_n_neighbors=20
)
```
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | `str \| Document` | Required | The search query |
| `n_results` | `int` | 5 | Number of results to return |
| `filter` | `Callable \| Dict \| None` | None | Optional filter for results |
| `fetch_n_neighbors` | `int` | 20 | Number of neighbors to fetch |
Source: [core/quivr_core/brain/brain.py:100-120](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
## Storage and Persistence
### Saving a Brain
Brains can be persisted to disk for later reuse:
```python
save_path = await brain.save("/home/user/.local/quivr")
```
Source: [examples/save_load_brain.py:1-22](https://github.com/QuivrHQ/quivr/blob/main/examples/save_load_brain.py)
### Loading a Brain
```python
brain_loaded = Brain.load(save_path)
brain_loaded.print_info()
```
Source: [examples/save_load_brain.py:18](https://github.com/QuivrHQ/quivr/blob/main/examples/save_load_brain.py)
## File Processing
### QuivrFile Entity
The Brain processes files through the `QuivrFile` dataclass:
```python
class QuivrFile:
__slots__ = [
"id",
"brain_id",
"path",
"original_filename",
"file_size",
"file_extension",
"file_sha1",
"additional_metadata",
]
```
Source: [core/quivr_core/files/file.py:20-35](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
### File Metadata
During processing, each chunk receives metadata including:
| Field | Description |
|-------|-------------|
| `chunk_index` | Position of the chunk in the document |
| `quivr_core_version` | Version of quivr-core used |
| `language` | Detected language of the content |
| `original_file_name` | Source filename |
Source: [core/quivr_core/processor/processor_base.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
## Retrieval Configuration
The `RetrievalConfig` controls the RAG pipeline behavior:
```python
from quivr_core.config import RetrievalConfig
config_file_name = "./basic_rag_workflow.yaml"
retrieval_config = RetrievalConfig.from_yaml(config_file_name)
```
### YAML Configuration Example
```yaml
workflow_config:
name: "standard RAG"
nodes:
- name: "START"
edges: ["filter_history"]
- name: "filter_history"
edges: ["rewrite"]
- name: "rewrite"
edges: [...]
```
## Complete Usage Example
```python
import tempfile
from quivr_core import Brain
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt") as temp_file:
temp_file.write("Gold is a liquid of blue-like colour.")
temp_file.flush()
brain = Brain.from_files(
name="test_brain",
file_paths=[temp_file.name],
)
answer = brain.ask("what is gold? answer in french")
print("answer:", answer)
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Attributes Summary
| Attribute | Type | Description |
|-----------|------|-------------|
| `id` | `UUID` | Unique identifier for the brain |
| `name` | `str` | Human-readable name |
| `vector_db` | `VectorStore` | Vector storage for embeddings |
| `llm` | `LLMEndpoint` | Language model for generation |
| `embedder` | `Embeddings` | Embedding model for vectorization |
| `storage` | `BrainStorage` | Persistence layer |
## Module Export
The Brain class is exported from the main `quivr_core` package:
```python
from quivr_core import Brain
```
Source: [core/quivr_core/brain/__init__.py:1-5](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/__init__.py)
---
## RAG Implementation
### Related Pages
Related topics: [Brain Class](#page-brain-class)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/rag/quivr_rag.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
- [core/quivr_core/rag/quivr_rag_langgraph.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag_langgraph.py)
- [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
- [core/quivr_core/rag/entities/chat.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/chat.py)
- [core/quivr_core/rag/prompts.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/prompts.py)
# RAG Implementation
## Overview
The RAG (Retrieval-Augmented Generation) implementation in Quivr provides a flexible, opinionated framework for building Retrieval-Augmented Generation pipelines. It combines vector-based document retrieval with Large Language Model (LLM) generation to enable question-answering over uploaded documents and files.
The implementation supports both synchronous and streaming responses, configurable retrieval strategies, and integration with multiple LLM providers including OpenAI, Anthropic, Mistral, and local models via Ollama.
## Architecture
```mermaid
graph TD
A[User Query] --> B[Brain.ask]
B --> C[RAG Pipeline]
C --> D[Retrieval Phase]
C --> E[Generation Phase]
D --> F[Vector DB Search]
F --> G[Relevant Chunks]
E --> H[LLM Processing]
G --> H
H --> I[Streaming Response]
I --> J[ParsedRAGChunkResponse]
```
## Core Components
### Brain Class
The `Brain` class serves as the main entry point for RAG operations. It orchestrates document processing, retrieval, and generation.
**Key Methods:**
| Method | Description | Return Type |
|--------|-------------|-------------|
| `from_files()` | Create a brain from file paths | `Brain` |
| `afrom_langchain_documents()` | Create a brain from LangChain documents | `Brain` |
| `ask()` | Synchronous question answering | `RAGResponse` |
| `asearch()` | Search for relevant documents | `list[SearchResult]` |
Source: [core/quivr_core/brain/brain.py:1-100](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
### RAG Pipeline Flow
```mermaid
sequenceDiagram
participant User
participant Brain
participant Retriever
participant VectorDB
participant LLM
participant Response
User->>Brain: ask(question)
Brain->>Retriever: retrieve(query)
Retriever->>VectorDB: similarity_search()
VectorDB-->>Retriever: relevant_chunks
Retriever-->>Brain: context_chunks
Brain->>LLM: generate(context, question)
LLM-->>Response: ParsedRAGChunkResponse
Response-->>User: Streaming Answer
```
## Streaming Response System
### ParsedRAGChunkResponse
The streaming response is built around the `ParsedRAGChunkResponse` data class, which provides incremental chunks of the generated answer along with metadata.
**Chunk Structure:**
| Field | Type | Description |
|-------|------|-------------|
| `answer` | `str` | The partial or complete answer text |
| `metadata` | `dict` | Sources and additional context |
| `last_chunk` | `bool` | Indicates final chunk of response |
| `sources` | `list` | Retrieved document sources |
| `chunk_id` | `int` | Sequence number of the chunk |
Source: [core/quivr_core/rag/quivr_rag.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
### Answer Streaming Implementation
The `answer_astream` method implements asynchronous streaming of LLM responses:
```python
async def answer_astream(self, query: str, ...) -> AsyncGenerator[ParsedRAGChunkResponse]:
# Processing logic yields ParsedRAGChunkResponse chunks
yield ParsedRAGChunkResponse(
answer="",
metadata=get_chunk_metadata(rolling_message, sources),
last_chunk=True,
)
```
**Streaming Characteristics:**
- Yields multiple `ParsedRAGChunkResponse` objects during generation
- Each chunk contains accumulated `rolling_message` content
- Uses `chunk_id` for tracking sequence order
- Final chunk marked with `last_chunk=True`
- Metadata includes retrieved sources for citation
Source: [core/quivr_core/rag/quivr_rag.py:50-100](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
## Retrieval Configuration
### RetrievalConfig
The retrieval behavior is controlled through `RetrievalConfig` which supports YAML-based configuration:
```yaml
workflow_config:
name: "standard RAG"
nodes:
- name: "START"
edges: ["filter_history"]
- name: "filter_history"
edges: ["rewrite"]
- name: "rewrite"
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
### Configuration Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `n_results` | `int` | `5` | Number of documents to retrieve |
| `fetch_n_neighbors` | `int` | `20` | Number of neighbors to fetch from vector DB |
| `filter` | `Callable \| Dict` | `None` | Optional metadata filtering |
Source: [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
## Search Operation
### Async Search Method
```python
async def asearch(
self,
query: str | Document,
n_results: int = 5,
filter: Callable | Dict[str, Any] | None = None,
fetch_n_neighbors: int = 20,
) -> list[SearchResult]:
```
**Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | `str \| Document` | Yes | The search query |
| `n_results` | `int` | No | Maximum results to return |
| `filter` | `Callable \| Dict` | No | Metadata filter condition |
| `fetch_n_neighbors` | `int` | No | Initial fetch size before re-ranking |
Source: [core/quivr_core/brain/brain.py:50-80](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
## Document Processing Pipeline
### Processor Base
The `processor_base.py` handles document chunking and metadata enrichment:
```python
async def process_file(file: QuivrFile) -> ProcessedDocument:
docs = await self.process_file_inner(file)
qvr_version = version("quivr-core")
for idx, doc in enumerate(docs.chunks, start=1):
doc.metadata = {
"chunk_index": idx,
"quivr_core_version": qvr_version,
"language": detect_language(text=...).value,
**file.metadata,
**doc.metadata,
}
```
**Metadata Enrichment:**
- `chunk_index`: Sequential position of chunk
- `quivr_core_version`: Version of quivr-core
- `language`: Auto-detected language of content
- Original filename embedded in content for reference
Source: [core/quivr_core/processor/processor_base.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
## Prompt Templates
### User Prompt Template
The prompt system uses structured templates with metadata injection:
```
{user_metadata}
{ticket_metadata}
{similar_tickets}
{ticket_history}
{additional_information}
{client_query}
```
Source: [core/quivr_core/rag/prompts.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/prompts.py)
### Default Instructions
The system includes default instructions that guide response generation:
| Instruction | Description |
|-------------|-------------|
| Conciseness | Use same level of detail as similar responses |
| Formatting | Proper paragraphs, bold, italic for readability |
| Language | Respond in same language as user query |
| Consistency | Maintain terminology consistency |
| No Signature | Signature added separately after response |
## LangGraph Integration
The implementation includes a LangGraph-based RAG pipeline (`quivr_rag_langgraph.py`) for more complex workflows:
```mermaid
graph LR
A[Query] --> B[History Filter]
B --> C[Query Rewrite]
C --> D[Retrieval]
D --> E[Answer Generation]
E --> F[Response]
```
Features:
- Multi-step processing pipelines
- Conversation history integration
- Query rewriting for better retrieval
- Customizable workflow nodes
Source: [core/quivr_core/rag/quivr_rag_langgraph.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag_langgraph.py)
## LLM Configuration
### LLMEndpointConfig
| Parameter | Type | Description |
|-----------|------|-------------|
| `model` | `str` | Model identifier |
| `llm_base_url` | `str` | API endpoint URL |
| `temperature` | `float` | Generation temperature (default: 0.7) |
Source: [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
### Supported Providers
- **OpenAI**: GPT-4, GPT-3.5-turbo
- **Anthropic**: Claude models
- **Mistral**: Mistral AI models
- **Ollama**: Local model support
## Usage Example
```python
from quivr_core import Brain
from quivr_core.config import RetrievalConfig
# Create brain from files
brain = Brain.from_files(
name="my_brain",
file_paths=["./document.pdf", "./notes.txt"],
)
# Configure retrieval
retrieval_config = RetrievalConfig.from_yaml("./workflow.yaml")
# Ask a question
answer = brain.ask(
"What is the main topic of these documents?",
retrieval_config=retrieval_config
)
print(answer.answer)
```
Source: [examples/simple_question/simple_question.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question/simple_question.py)
## Key Design Patterns
| Pattern | Implementation |
|---------|-----------------|
| Async/Await | All I/O operations are asynchronous |
| Streaming | Responses streamed via async generators |
| Dependency Injection | LLM and embedder are configurable |
| Configuration-driven | Workflows defined via YAML |
| Metadata Enrichment | Automatic language detection and versioning |
---
## LLM Integration
### Related Pages
Related topics: [Core Components](#page-core-components), [System Architecture Overview](#page-architecture-overview)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/llm/llm_endpoint.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/llm/llm_endpoint.py)
- [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
- [core/quivr_core/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/config.py)
- [core/quivr_core/base_config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/base_config.py)
- [core/quivr_core/rag/quivr_rag_langgraph.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag_langgraph.py)
# LLM Integration
## Overview
The LLM Integration module in Quivr provides a flexible, abstracted interface for interacting with Large Language Models (LLMs) from various providers. This module enables Quivr's RAG (Retrieval-Augmented Generation) pipeline to leverage different LLM backends without requiring changes to the core business logic. The integration supports OpenAI, Anthropic, Mistral, Meta (Llama), Groq, and local models via Ollama.
The primary goals of the LLM Integration are:
- **Provider Abstraction**: Uniform API regardless of the underlying LLM provider
- **Configuration Management**: Centralized configuration for model parameters, token limits, and provider-specific settings
- **Runtime Flexibility**: Support for both synchronous and asynchronous operations
- **Embeddings Integration**: Seamless integration with embedding models for semantic search
Source: [core/quivr_core/llm/llm_endpoint.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/llm/llm_endpoint.py)
## Architecture
### High-Level Components
The LLM Integration consists of several key components:
| Component | Purpose | Location |
|-----------|---------|----------|
| `LLMEndpoint` | Main wrapper class for LLM interactions | `quivr_core/llm/llm_endpoint.py` |
| `LLMEndpointConfig` | Configuration dataclass for LLM settings | `quivr_core/rag/entities/config.py` |
| `LLMConfig` | Per-model configuration with token limits | `quivr_core/rag/entities/config.py` |
| `DefaultModelSuppliers` | Enum for supported LLM providers | `quivr_core/rag/entities/config.py` |
### Class Diagram
```mermaid
classDiagram
class LLMEndpoint {
+llm: BaseChatModel
+llm_config: LLMEndpointConfig
+__init__(llm, llm_config)
+get_client() BaseChatModel
}
class LLMEndpointConfig {
+model: str
+llm_base_url: str
+temperature: float
+max_output_tokens: int
+model_type: LLMEndpointType
}
class LLMConfig {
+max_context_tokens: int
+max_output_tokens: int
+tokenizer_hub: str
}
class DefaultModelSuppliers {
<>
OPENAI
ANTHROPIC
MISTRAL
META
GROQ
OLLAMA
}
LLMEndpoint --> LLMEndpointConfig
LLMEndpointConfig ..> DefaultModelSuppliers
```
Source: [core/quivr_core/llm/llm_endpoint.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/llm/llm_endpoint.py)
Source: [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
## Configuration
### LLMEndpointConfig Parameters
The `LLMEndpointConfig` class provides configuration for LLM endpoints:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `model` | `str` | Required | Model identifier (e.g., "gpt-4o", "claude-3-opus") |
| `llm_base_url` | `str` | `"local"` | Base URL for the LLM API endpoint |
| `temperature` | `float` | `0.7` | Sampling temperature for generation (0.0-2.0) |
| `max_output_tokens` | `int` | `4096` | Maximum tokens in the generated response |
| `model_type` | `LLMEndpointType` | `LLMEndpointType.CHAT` | Type of LLM endpoint |
Source: [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
### Default Model Configurations
Quivr provides pre-configured settings for various models through the `DefaultModelSuppliers` enum and associated `LLMConfig` dictionaries:
| Provider | Model | Max Context Tokens | Max Output Tokens | Tokenizer Hub |
|----------|-------|-------------------|-------------------|---------------|
| **OpenAI** | gpt-4o | 128,000 | 16,384 | Quivr/claude-tokenizer |
| **OpenAI** | gpt-4-turbo | 128,000 | 4,096 | Quivr/claude-tokenizer |
| **Anthropic** | claude-3.5-sonnet | 200,000 | 8,192 | Quivr/claude-tokenizer |
| **Anthropic** | claude-3-opus | 200,000 | 4,096 | Quivr/claude-tokenizer |
| **Mistral** | mistral-large | 32,000 | N/A | - |
| **Meta** | llama-3.1 | 128,000 | 4,096 | Quivr/Meta-Llama-3.1-Tokenizer |
| **Meta** | llama-3 | 8,192 | 2,048 | Quivr/llama3-tokenizer-new |
| **Groq** | llama-3.3-70b | 128,000 | 32,768 | Quivr/Meta-Llama-3.1-Tokenizer |
Source: [core/quivr_core/rag/entities/config.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/entities/config.py)
### Environment Variables
API keys should be set as environment variables before initializing the LLM:
```bash
export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
```
Example usage in code:
```python
import os
os.environ["OPENAI_API_KEY"] = "myopenai_apikey"
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## Usage Patterns
### Basic Integration with Brain
The most common pattern is to pass an `LLMEndpoint` instance when creating a Brain:
```python
from langchain_openai import ChatOpenAI
from quivr_core import Brain
from quivr_core.llm.llm_endpoint import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
brain = Brain.from_files(
name="my_smart_brain",
file_paths=["./documents/*.pdf"],
llm=LLMEndpoint(
llm_config=LLMEndpointConfig(model="gpt-4o"),
llm=ChatOpenAI(model="gpt-4o", api_key=str(os.getenv("OPENAI_API_KEY"))),
),
)
```
Source: [examples/simple_question_megaparse.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question_megaparse.py)
### Using Fake LLM for Testing
For testing purposes, Quivr supports fake LLM implementations:
```python
from langchain_core.language_models import FakeListChatModel
from quivr_core import Brain
from quivr_core.llm.llm_endpoint import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
from langchain_core.embeddings import DeterministicFakeEmbedding
brain = Brain.from_files(
name="test_brain",
file_paths=["tests/processor/data/dummy.pdf"],
llm=LLMEndpoint(
llm=FakeListChatModel(responses=["good"]),
llm_config=LLMEndpointConfig(model="fake_model", llm_base_url="local"),
),
embedder=DeterministicFakeEmbedding(size=20),
)
```
Source: [examples/pdf_parsing_tika.py](https://github.com/QuivrHQ/quivr/blob/main/examples/pdf_parsing_tika.py)
### Custom Embeddings Integration
When using custom LLM configurations, you can also specify custom embedders:
```python
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from quivr_core import Brain
from quivr_core.llm.llm_endpoint import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
# LLM Configuration
llm = LLMEndpoint(
llm_config=LLMEndpointConfig(model="gpt-4o"),
llm=ChatOpenAI(model="gpt-4o", api_key=str(os.getenv("OPENAI_API_KEY"))),
)
# Embedder Configuration
embedder = OpenAIEmbeddings(model="text-embedding-3-large")
brain = Brain.from_files(
name="custom_brain",
file_paths=["./data/documents/"],
llm=llm,
embedder=embedder,
)
```
Source: [examples/simple_question_megaparse.py](https://github.com/QuivrHQ/quivr/blob/main/examples/simple_question_megaparse.py)
## Supported Providers
### OpenAI
OpenAI models are supported through the `langchain_openai` package:
```python
from langchain_openai import ChatOpenAI
from quivr_core.llm.llm_endpoint import LLMEndpoint
from quivr_core.rag.entities.config import LLMEndpointConfig
llm = LLMEndpoint(
llm=ChatOpenAI(model="gpt-4o", api_key=api_key),
llm_config=LLMEndpointConfig(
model="gpt-4o",
temperature=0.7,
),
)
```
Supported models: `gpt-4o`, `gpt-4-turbo`, `gpt-3.5-turbo`
### Anthropic
Anthropic models require the `langchain-anthropic` package:
```python
from langchain_anthropic import ChatAnthropic
from quivr_core.llm.llm_endpoint import LLMEndpoint
llm = LLMEndpoint(
llm=ChatAnthropic(model="claude-3-5-sonnet-20241022", anthropic_api_key=api_key),
llm_config=LLMEndpointConfig(model="claude-3.5-sonnet"),
)
```
### Mistral
Mistral models are supported via their API:
```python
from langchain_mistralai import ChatMistralAI
llm = LLMEndpoint(
llm=ChatMistralAI(model="mistral-large-latest", mistral_api_key=api_key),
llm_config=LLMEndpointConfig(model="mistral-large"),
)
```
### Local Models (Ollama)
For local inference using Ollama:
```python
from langchain_ollama import ChatOllama
llm = LLMEndpoint(
llm=ChatOllama(model="llama3.1", base_url="http://localhost:11434"),
llm_config=LLMEndpointConfig(model="llama-3.1", llm_base_url="http://localhost:11434"),
)
```
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## RAG Workflow Integration
The LLM Integration is a core component of Quivr's RAG pipeline. When processing a user query, the LLM is responsible for generating the final answer based on retrieved context.
```mermaid
graph TD
A[User Query] --> B[Brain.ask]
B --> C[Vector Search]
C --> D[Retrieve Relevant Chunks]
D --> E[LLMEndpoint]
E --> F[Generate Answer]
F --> G[RAG Response]
H[LLMEndpointConfig] --> E
I[Chat History] --> E
J[System Prompt] --> E
```
The LLM receives contextual information from the retrieval step and generates responses that are then formatted and returned to the user. The `quivr_rag_langgraph.py` module orchestrates this workflow using LangGraph for complex graph-based processing.
Source: [core/quivr_core/rag/quivr_rag.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag.py)
Source: [core/quivr_core/rag/quivr_rag_langgraph.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/rag/quivr_rag_langgraph.py)
## Default LLM Fallback
If no LLM is explicitly provided when creating a Brain, Quivr automatically initializes a default LLM:
```python
# From brain.py source code
if llm is None:
llm = default_llm()
```
This fallback mechanism ensures that users can get started quickly without explicit configuration, while still allowing advanced users to customize their LLM setup.
Source: [core/quivr_core/brain/brain.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/brain/brain.py)
## Best Practices
### API Key Security
- Store API keys in environment variables, never hardcode them
- Use `.env` files with proper `.gitignore` entries for local development
- Consider using secret management services (AWS Secrets Manager, HashiCorp Vault) for production
### Model Selection
- Choose models based on your use case requirements (speed vs. quality)
- Use smaller models for simple queries to reduce costs
- Reserve larger models (e.g., GPT-4o, Claude 3.5) for complex reasoning tasks
### Token Management
- Monitor `max_context_tokens` to avoid exceeding model limits
- Set appropriate `max_output_tokens` based on expected response length
- Use the `RetrievalConfig` to control how many chunks are fed to the LLM
### Temperature Settings
| Temperature | Use Case | Characteristics |
|-------------|----------|-----------------|
| 0.0 - 0.3 | Factual/Deterministic | More focused, consistent responses |
| 0.4 - 0.7 | General Purpose | Balanced creativity and consistency |
| 0.8 - 1.0 | Creative/Brainstorming | More varied, potentially surprising outputs |
## Troubleshooting
### Common Issues
1. **API Key Not Found**
```
Error: OPENAI_API_KEY environment variable not set
```
Solution: Ensure the API key is set before running the script:
```bash
export OPENAI_API_KEY="your-key"
```
2. **Model Context Limit Exceeded**
Solution: Reduce the number of retrieved chunks or adjust chunk size in processing
3. **Rate Limiting**
Solution: Implement retry logic or use a rate limiting middleware
Source: [README.md](https://github.com/QuivrHQ/quivr/blob/main/README.md)
## See Also
- [Brain Configuration](./brain-configuration.md)
- [RAG Workflows](./rag-workflows.md)
- [Embedding Integration](./embedding-integration.md)
- [Quivr Documentation](https://core.quivr.com/)
---
## File Processing and Parsers
### Related Pages
Related topics: [Storage System](#page-storage)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/processor/processor_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
- [core/quivr_core/processor/registry.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/registry.py)
- [core/quivr_core/processor/implementations/default.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/default.py)
- [core/quivr_core/processor/implementations/simple_txt_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
- [core/quivr_core/processor/implementations/tika_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/tika_processor.py)
- [core/quivr_core/processor/implementations/megaparse_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/megaparse_processor.py)
# File Processing and Parsers
Quivr's file processing system provides a extensible architecture for ingesting, parsing, and chunking various document formats for use in Retrieval-Augmented Generation (RAG) workflows. The processor subsystem sits at the core of Quivr's data ingestion pipeline, transforming raw files into structured document chunks ready for embedding and retrieval.
## Architecture Overview
The file processing architecture follows a plugin-based design pattern where processors are registered by file extension and lazily loaded on demand. This separation of concerns allows new file format support to be added without modifying core processing logic.
```mermaid
graph TD
A[QuivrFile Input] --> B[Processor Registry]
B --> C{File Extension}
C -->|.txt| D[SimpleTxtProcessor]
C -->|.pdf| E[MegaparseProcessor]
C -->|Other| F[Default Processors]
D --> G[ProcessedDocument]
E --> G
F --> G
G --> H[Document Chunks]
H --> I[RAG Pipeline]
```
### Core Components
| Component | Purpose | Location |
|-----------|---------|----------|
| `ProcessorBase` | Abstract base class for all processors | `processor_base.py` |
| `ProcessedDocument` | Generic container for processing results | `processor_base.py` |
| `ProcessorRegistry` | Maps extensions to processor classes | `registry.py` |
| `SplitterConfig` | Configuration for text chunking | `splitter.py` |
Source: [processor_base.py:1-75](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py), [registry.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/registry.py)
---
## ProcessorBase Abstract Class
The `ProcessorBase` serves as the foundation for all file processors in Quivr. It defines the contract that every processor implementation must follow.
### Class Definition
```python
class ProcessorBase(ABC, Generic[R]):
supported_extensions: list[FileExtension | str]
@property
@abstractmethod
def processor_metadata(self) -> dict[str, Any]:
raise NotImplementedError
async def process_file(self, file: QuivrFile) -> ProcessedDocument[R]:
...
@abstractmethod
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[R]:
raise NotImplementedError
```
Source: [processor_base.py:47-75](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
### Key Methods
| Method | Access | Description |
|--------|--------|-------------|
| `supported_extensions` | Class attribute | List of file extensions the processor handles |
| `check_supported()` | Instance | Validates if a file type is supported |
| `process_file()` | Async | Main entry point; validates, processes, and enriches metadata |
| `process_file_inner()` | Abstract Async | Subclass implementation for actual parsing logic |
| `processor_metadata` | Abstract Property | Returns processor configuration as dict |
### Metadata Enrichment
The `process_file()` method automatically enriches each document chunk with standardized metadata:
```python
doc.metadata = {
"chunk_index": idx,
"quivr_core_version": qvr_version,
"language": detect_language(...).value,
**file.metadata,
**doc.metadata,
**self.processor_metadata,
}
```
This ensures all chunks carry provenance information including the Quivr version, detected language, and processor configuration.
Source: [processor_base.py:20-39](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
---
## ProcessedDocument Generic Container
The `ProcessedDocument` is a generic wrapper that encapsulates the output of any processor:
```python
@dataclass
class ProcessedDocument(Generic[R]):
chunks: List[Document] # List of LangChain Document objects
processor_cls: str # Name of the processor class
processor_response: R # Processor-specific return value
```
The generic type `R` allows each processor to define its own response type. For example, `SimpleTxtProcessor` returns `ProcessedDocument[str]` where `processor_response` contains the raw text content.
Source: [processor_base.py:40-46](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
---
## Text Chunking and Splitting
Quivr provides configurable text chunking through the `SplitterConfig` class and custom splitting implementations.
### SplitterConfig
```python
@dataclass
class SplitterConfig(BaseModel):
chunk_size: int = 500
chunk_overlap: int = 0
length_function: Callable[[str], int] = len
```
| Parameter | Default | Description |
|-----------|---------|-------------|
| `chunk_size` | 500 | Maximum characters per chunk |
| `chunk_overlap` | 0 | Characters to overlap between chunks |
| `length_function` | `len` | Function to calculate text length |
### SimpleTxtProcessor Recursive Splitter
The `SimpleTxtProcessor` implements a custom recursive character splitter:
```python
def recursive_character_splitter(
doc: Document, chunk_size: int, chunk_overlap: int
) -> list[Document]:
if len(doc.page_content) <= chunk_size:
return [doc]
chunk = Document(page_content=doc.page_content[:chunk_size], metadata=doc.metadata)
remaining = Document(
page_content=doc.page_content[chunk_size - chunk_overlap:],
metadata=doc.metadata,
)
return [chunk] + recursive_character_splitter(remaining, chunk_size, chunk_overlap)
```
**Constraint**: `chunk_overlap` must be less than `chunk_size`
Source: [simple_txt_processor.py:8-26](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
### LangChain Text Splitters
Other processors leverage LangChain's `RecursiveCharacterTextSplitter`:
```python
from langchain_text_splitters import RecursiveCharacterTextSplitter, TextSplitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=splitter_config.chunk_size,
chunk_overlap=splitter_config.chunk_overlap,
length_function=tiktoken_len, # Token-based length
)
docs = text_splitter.split_documents([document])
```
Source: [megaparse_processor.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/megaparse_processor.py)
---
## Available Processors
### Processor Comparison
| Processor | File Types | Parser Backend | Notes |
|-----------|------------|----------------|-------|
| `SimpleTxtProcessor` | `.txt` | Custom recursive splitter | Lightweight, no external dependencies |
| `MegaparseProcessor` | `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.epub`, `.bib`, `.odt`, `.html`, `.md`, `.mdx` | MegaParse SDK | AI-optimized document parsing |
| `TikaProcessor` | `.pdf` | Apache Tika | Robust PDF extraction |
| Default Processors | Various | LangChain loaders | Catch-all for standard formats |
### SimpleTxtProcessor
Designed for plain text files with zero external dependencies:
```python
class SimpleTxtProcessor(ProcessorBase):
supported_extensions = [FileExtension.txt]
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[str]:
async with aiofiles.open(file.path, mode="r") as f:
content = await f.read()
doc = Document(page_content=content)
docs = recursive_character_splitter(
doc, self.splitter_config.chunk_size, self.splitter_config.chunk_overlap
)
return ProcessedDocument(
chunks=docs,
processor_cls="SimpleTxtProcessor",
processor_response=content
)
```
Source: [simple_txt_processor.py:30-60](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/simple_txt_processor.py)
### MegaparseProcessor
The most versatile processor, supporting 14+ file formats through the MegaParse SDK:
```python
class MegaparseProcessor(ProcessorBase[MPDocument]):
supported_extensions = [
FileExtension.txt, FileExtension.pdf, FileExtension.docx,
FileExtension.pptx, FileExtension.xlsx, FileExtension.csv,
FileExtension.epub, FileExtension.html, FileExtension.markdown,
# ... more extensions
]
def __init__(
self,
splitter: TextSplitter | None = None,
splitter_config: SplitterConfig = SplitterConfig(),
megaparse_config: MegaparseConfig = MegaparseConfig(),
) -> None:
...
```
**Installation**: `pip install megaparse`
Source: [megaparse_processor.py:1-60](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/megaparse_processor.py)
### Default Processors
Built using LangChain's community loaders for standard formats:
| LangChain Loader | File Extension |
|------------------|----------------|
| `UnstructuredPDFLoader` | `.pdf` |
| `Docx2txtLoader` | `.docx` |
| `UnstructuredPowerPointLoader` | `.pptx` |
| `UnstructuredExcelLoader` | `.xls`, `.xlsx` |
| `CSVLoader` | `.csv` |
| `UnstructuredEPubLoader` | `.epub` |
| `UnstructuredMarkdownLoader` | `.md`, `.markdown` |
| `UnstructuredHTMLLoader` | `.html` |
| `TextLoader` | `.txt` |
These are dynamically generated using `_build_processor()` factory:
```python
def _build_processor(
cls_name: str,
load_cls: Type[P],
cls_extensions: List[FileExtension | str]
) -> Type[ProcessorInit]:
enc = tiktoken.get_encoding("cl100k_base")
class _Processor(ProcessorBase):
supported_extensions = cls_extensions
# ... implementation
return _Processor
```
Source: [default.py:1-60](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/default.py)
### TikaProcessor
Uses Apache Tika for robust PDF extraction with token-based chunk sizing:
```python
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[None]:
async with file.open() as f:
txt = await self._send_parse_tika(f)
document = Document(page_content=txt)
docs = self.text_splitter.split_documents([document])
for doc in docs:
doc.metadata = {"chunk_size": len(self.enc.encode(doc.page_content))}
return ProcessedDocument(
chunks=docs,
processor_cls="TikaProcessor",
processor_response=None
)
```
Source: [tika_processor.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/implementations/tika_processor.py)
---
## Processor Registry
The registry provides lazy-loading lookup from file extensions to processor classes:
```python
def get_processor_class(file_extension: FileExtension | str) -> Type[ProcessorBase]:
"""Fetch processor class from registry
Loading of these classes is *Lazy*. Appropriate import will happen
the first time we try to process some file type.
"""
...
return known_processors[file_extension].cls
```
### Registry Structure
```python
ProcEntry = namedtuple('ProcEntry', ['exts', 'cls_mod', 'errtxt', 'priority'])
known_processors = defaults_to_proc_entries(base_processors)
```
Each entry contains:
- `exts`: List of file extensions
- `cls_mod`: Module path to import
- `errtxt`: Error message if import fails
- `priority`: Processor priority (for multiple handlers)
### Registration Flow
```mermaid
graph LR
A[defaults_to_proc_entries] --> B[_append_proc_mapping]
B --> C[base_processors dict]
C --> D[known_processors]
D --> E[get_processor_class]
E --> F{Lazy Import}
F --> G[Processor Class]
```
Source: [registry.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/registry.py)
---
## Processing Workflow
```mermaid
sequenceDiagram
participant User
participant Brain
participant Registry
participant Processor
participant Splitter
User->>Brain: add_files(file_paths)
Brain->>Registry: get_processor_class(extension)
Registry-->>Brain: Processor Class
Brain->>Processor: process_file(QuivrFile)
Processor->>Processor: check_supported()
Processor->>Processor: process_file_inner()
Processor->>Splitter: split_documents()
Splitter-->>Processor: List[Document]
Processor->>Processor: enrich_metadata()
Processor-->>Brain: ProcessedDocument
Brain->>Brain: vectorstore.add_documents()
```
### Step-by-Step Processing
1. **File Input**: `QuivrFile` object created from file path with metadata (SHA1, size, extension)
2. **Registry Lookup**: Extension matched to appropriate processor class
3. **Validation**: `check_supported()` confirms file type compatibility
4. **Parsing**: `process_file_inner()` extracts raw text/content
5. **Chunking**: Documents split according to `SplitterConfig`
6. **Metadata Enrichment**: Each chunk receives index, version, and language tags
7. **Output**: `ProcessedDocument` returned with chunks ready for embedding
---
## Configuration Options
### Splitter Configuration
```python
from quivr_core.processor.splitter import SplitterConfig
config = SplitterConfig(
chunk_size=1000, # Characters per chunk
chunk_overlap=100, # Overlap between chunks
length_function=len # Or tiktoken-based for token counting
)
```
### Megaparse Configuration
```python
from quivr_core.config import MegaparseConfig
config = MegaparseConfig(
# Provider-specific settings
)
```
---
## Extending with Custom Processors
To add support for a new file format:
1. Create a class extending `ProcessorBase[R]`
2. Implement `process_file_inner()` method
3. Define `supported_extensions` class attribute
4. Implement `processor_metadata` property
5. Register in `registry.py` or dynamically
```python
class CustomProcessor(ProcessorBase[MyResponseType]):
supported_extensions = [FileExtension.my_format]
@property
def processor_metadata(self) -> dict[str, Any]:
return {"processor_cls": "CustomProcessor"}
async def process_file_inner(self, file: QuivrFile) -> ProcessedDocument[MyResponseType]:
# Custom parsing logic
...
return ProcessedDocument(chunks=docs, processor_cls="CustomProcessor", processor_response=result)
```
---
## Error Handling
Processors validate file support at runtime:
```python
def check_supported(self, file: QuivrFile) -> None:
if file.file_extension not in self.supported_extensions:
raise ValueError(f"can't process a file of type {file.file_extension}")
```
Lazy loading provides graceful degradation for optional dependencies:
```python
try:
from megaparse_sdk.client import MegaParseNATSClient
except ImportError:
raise ImportError("Please install megaparse: pip install megaparse")
```
Source: [processor_base.py:50-53](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/processor/processor_base.py)
---
## Related Components
- **QuivrFile**: Represents an input file with metadata (path, SHA1, size, extension)
- **Brain**: Orchestrates file processing and retrieval
- **VectorStore**: Stores processed document chunks for similarity search
- **RetrievalConfig**: Configures how chunks are retrieved during Q&A
---
## Storage System
### Related Pages
Related topics: [File Processing and Parsers](#page-file-processing), [Brain Class](#page-brain-class)
Relevant source files
The following source files were used to generate this page:
- [core/quivr_core/storage/storage_base.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/storage_base.py)
- [core/quivr_core/storage/local_storage.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/local_storage.py)
- [core/quivr_core/storage/file.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/file.py)
- [core/quivr_core/files/file.py](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
# Storage System
## Overview
The Storage System in Quivr is a modular abstraction layer responsible for managing file uploads, storage, and retrieval across different storage backends. It provides a clean separation between the RAG (Retrieval-Augmented Generation) logic and the underlying file persistence mechanisms, enabling Quivr to work with various storage implementations while maintaining a consistent API.
The system is designed to handle:
- File upload and deduplication using SHA-1 hashing
- Asynchronous file access
- Multiple storage backends (currently supporting local filesystem)
- File metadata tracking and organization by brain
Source: [core/quivr_core/storage/storage_base.py:1-47](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/storage_base.py)
## Architecture
### High-Level Architecture
```mermaid
graph TD
A[Brain] --> B[StorageBase]
B --> C[LocalStorage]
B --> D[Future: CloudStorage]
B --> E[Future: S3Storage]
C --> F[QuivrFile]
C --> G[File System]
F --> H[Metadata]
F --> I[SHA-1 Hash]
```
### Component Hierarchy
```mermaid
classDiagram
class StorageBase {
<>
+name: str
+nb_files() int
+get_files() List~QuivrFile~
+upload_file(file, exists_ok)*
}
class LocalStorage {
+name: str = "local_storage"
+files: List~QuivrFile~
+hashes: Set~str~
+copy_flag: bool
+dir_path: Path
}
class QuivrFile {
+id: UUID
+brain_id: UUID
+path: Path
+original_filename: str
+file_size: int
+file_extension: FileExtension
+file_sha1: str
+additional_metadata: dict
}
StorageBase <|-- LocalStorage
```
## Core Components
### StorageBase (Abstract Base Class)
The `StorageBase` is an abstract base class that defines the contract for all storage implementations. It enforces a consistent interface across different storage backends through Python's ABC (Abstract Base Class) mechanism.
Source: [core/quivr_core/storage/storage_base.py:19-47](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/storage_base.py)
#### Required Subclass Attributes
| Attribute | Type | Description |
|-----------|------|-------------|
| `name` | `str` | Human-readable name of the storage type |
#### Abstract Methods
| Method | Return Type | Description |
|--------|-------------|-------------|
| `nb_files()` | `int` | Returns the total number of files stored |
| `get_files()` | `List[QuivrFile]` | Asynchronously retrieves all files in storage |
| `upload_file(file, exists_ok)` | `None` | Uploads a file to the storage |
The base class enforces that subclasses must define the `name` attribute through `__init_subclass__`:
```python
def __init_subclass__(cls, **kwargs):
for required in ("name",):
if not getattr(cls, required):
raise TypeError(
f"Can't instantiate abstract class {cls.__name__} without {required} attribute defined"
)
return super().__init_subclass__(**kwargs)
```
Source: [core/quivr_core/storage/storage_base.py:19-27](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/storage_base.py)
### LocalStorage Implementation
`LocalStorage` is the concrete implementation of `StorageBase` that persists files to the local filesystem.
Source: [core/quivr_core/storage/local_storage.py:1-50](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/local_storage.py)
#### Attributes
| Attribute | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | `str` | `"local_storage"` | Storage type identifier |
| `files` | `List[QuivrFile]` | `[]` | In-memory list of stored files |
| `hashes` | `Set[str]` | `set()` | Set of SHA-1 hashes for deduplication |
| `copy_flag` | `bool` | `True` | If `True`, copy files; if `False`, create symlinks |
| `dir_path` | `Path` | See below | Directory path for file storage |
#### Directory Resolution
The storage directory is resolved in the following order:
1. Explicitly provided `dir_path` argument
2. Environment variable `QUIVR_LOCAL_STORAGE`
3. Default path: `~/.cache/quivr/files`
```python
if dir_path is None:
self.dir_path = Path(
os.getenv("QUIVR_LOCAL_STORAGE", "~/.cache/quivr/files")
)
else:
self.dir_path = dir_path
os.makedirs(self.dir_path, exist_ok=True)
```
Source: [core/quivr_core/storage/local_storage.py:38-46](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/local_storage.py)
### QuivrFile Data Model
`QuivrFile` represents a file stored in the Quivr system with all associated metadata.
Source: [core/quivr_core/files/file.py:48-74](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
#### Class Slots
The class uses `__slots__` for memory-efficient attribute storage:
| Slot | Type | Description |
|------|------|-------------|
| `id` | `UUID` | Unique identifier for the file |
| `brain_id` | `UUID \| None` | ID of the brain this file belongs to |
| `path` | `Path` | Actual filesystem path to the file |
| `original_filename` | `str` | Original name of the uploaded file |
| `file_size` | `int \| None` | Size of the file in bytes |
| `file_extension` | `FileExtension \| str` | File extension/type |
| `file_sha1` | `str` | SHA-1 hash of the file content |
| `additional_metadata` | `dict` | Custom metadata dictionary |
Source: [core/quivr_core/files/file.py:49-57](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
#### Constructor Parameters
```python
def __init__(
self,
id: UUID,
original_filename: str,
path: Path,
file_sha1: str,
file_extension: FileExtension | str,
brain_id: UUID | None = None,
file_size: int | None = None,
metadata: dict[str, Any] | None = None,
) -> None:
```
Source: [core/quivr_core/files/file.py:59-70](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
#### Async File Access
The `QuivrFile` class provides an async context manager for reading file contents:
```python
@asynccontextmanager
async def open(self) -> AsyncGenerator[AsyncIterable[bytes], None]:
f = await aiofiles.open(self.path, mode="rb")
try:
yield f
finally:
await f.close()
```
Source: [core/quivr_core/files/file.py:76-83](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
## File Upload Workflow
```mermaid
sequenceDiagram
participant Client
participant LocalStorage
participant FileSystem
Client->>LocalStorage: upload_file(QuivrFile)
LocalStorage->>LocalStorage: Check SHA-1 hash
alt file exists and exists_ok=False
LocalStorage-->>Client: FileExistsError
else file doesn't exist
LocalStorage->>FileSystem: Copy or Symlink
FileSystem-->>LocalStorage: Success
LocalStorage->>LocalStorage: Update files list
LocalStorage->>LocalStorage: Add hash to hashes set
end
```
### Upload Process Details
1. **File Path Construction**: Files are stored at `{dir_path}/{brain_id}/{file_id}`
2. **Deduplication**: Before upload, the system checks if a file with the same SHA-1 hash already exists using `file_sha1`:
```python
if file.file_sha1 in self.hashes and not exists_ok:
raise FileExistsError(...)
```
3. **Storage Strategy**: Based on `copy_flag`:
- `True`: Copy file content to destination
- `False`: Create symbolic link to original file
4. **File Registration**: After successful upload, the file is added to the internal tracking list.
Source: [core/quivr_core/storage/local_storage.py:48-70](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/local_storage.py)
## File Creation Helper
The `get_file_path()` function creates a `QuivrFile` instance from a filesystem path:
```python
def get_file_path(
path: Path,
brain_id: UUID | None = None
) -> QuivrFile:
```
### Processing Steps
| Step | Operation |
|------|-----------|
| 1 | Get file size using `os.path.getsize()` |
| 2 | Compute SHA-1 hash by reading file bytes |
| 3 | Extract or generate UUID for file ID |
| 4 | Determine file extension |
| 5 | Create and return `QuivrFile` instance |
```python
file_size = os.path.getsize(path)
with aiofiles.open(path, mode="rb") as f:
file_sha1 = hashlib.sha1(await f.read()).hexdigest()
try:
id = UUID(path.name)
except ValueError:
id = uuid4()
```
Source: [core/quivr_core/storage/file.py:19-43](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/storage/file.py)
## Configuration Options
### LocalStorage Configuration
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `dir_path` | `Path \| None` | `None` | Custom storage directory |
| `copy_flag` | `bool` | `True` | File storage method |
### Environment Variables
| Variable | Description |
|----------|-------------|
| `QUIVR_LOCAL_STORAGE` | Override default storage directory |
## Usage Examples
### Creating a LocalStorage Instance
```python
from quivr_core.storage.local_storage import LocalStorage
from pathlib import Path
# Use default directory (~/.cache/quivr/files)
storage = LocalStorage()
# Use custom directory
storage = LocalStorage(dir_path=Path("/my/custom/storage"))
# Use symlinks instead of copying
storage = LocalStorage(dir_path=Path("/mnt/data"), copy_flag=False)
```
### Checking Storage Information
```python
from quivr_core.storage.local_storage import LocalStorage
storage = LocalStorage()
# Get number of files
file_count = storage.nb_files()
# Get storage info
info = storage.info()
# Returns: {"directory_path": "...", "name": "local_storage", "nb_files": N}
```
### Uploading Files
```python
import asyncio
from quivr_core.storage.local_storage import LocalStorage
from quivr_core.files.file import get_file_path
from pathlib import Path
async def upload_example():
storage = LocalStorage()
# Create QuivrFile from path
qfile = get_file_path(Path("./document.pdf"), brain_id=None)
# Upload with overwrite allowed
await storage.upload_file(qfile, exists_ok=True)
print(f"Total files: {storage.nb_files()}")
```
### Listing All Files
```python
import asyncio
from quivr_core.storage.local_storage import LocalStorage
async def list_files():
storage = LocalStorage()
files = await storage.get_files()
for f in files:
print(f"File: {f.original_filename}")
print(f" ID: {f.id}")
print(f" Size: {f.file_size} bytes")
print(f" SHA-1: {f.file_sha1}")
```
## Memory Management
The `QuivrFile` class uses `__slots__` to optimize memory usage by restricting attribute creation:
```python
class QuivrFile:
__slots__ = [
"id",
"brain_id",
"path",
"original_filename",
"file_size",
"file_extension",
"file_sha1",
"additional_metadata",
]
```
This approach:
- Prevents arbitrary attribute assignment
- Reduces memory overhead per instance
- Improves attribute access speed
Source: [core/quivr_core/files/file.py:48-57](https://github.com/QuivrHQ/quivr/blob/main/core/quivr_core/files/file.py)
## Extending the Storage System
To create a custom storage backend, implement the `StorageBase` abstract class:
```python
from quivr_core.storage.storage_base import StorageBase
from quivr_core.files.file import QuivrFile
class MyCustomStorage(StorageBase):
name = "my_custom_storage"
def __init__(self):
self._files = []
def nb_files(self) -> int:
return len(self._files)
async def get_files(self) -> list[QuivrFile]:
return self._files
async def upload_file(self, file: QuivrFile, exists_ok: bool = False) -> None:
# Custom implementation
pass
```
## Best Practices
1. **Deduplication**: Always compute and check SHA-1 hashes before uploading to avoid duplicate files.
2. **Async Operations**: All file I/O operations are asynchronous. Use `await` properly in async contexts.
3. **Memory Efficiency**: Use `__slots__` for custom file classes to reduce memory footprint.
4. **Path Handling**: Use `pathlib.Path` for cross-platform path operations.
5. **Error Handling**: Handle `FileExistsError` when uploading files with `exists_ok=False` (default).
6. **Directory Management**: Ensure storage directories exist before operations using `os.makedirs(path, exist_ok=True)`.
## Related Components
| Component | Description |
|-----------|-------------|
| [Brain](../brain/brain.md) | Uses Storage for file management |
| [Processor System](../processor/processor.md) | Processes files from storage |
| [RAG System](../rag/rag.md) | Retrieves documents from storage for query answering |
---
---
## Doramagic Pitfall Log
Project: QuivrHQ/quivr
Summary: Found 17 potential pitfall items; 2 are high/blocking. Highest priority: security_permissions - 来源证据：EU AI Act Compliance Scan Results — Sharing Findings for Feedback.
## 1. security_permissions · 来源证据：EU AI Act Compliance Scan Results — Sharing Findings for Feedback
- Severity: high
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：EU AI Act Compliance Scan Results — Sharing Findings for Feedback
- User impact: 可能影响授权、密钥配置或安全边界。
- Suggested check: 来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_46118108af95480ba852109be6e2c66a | https://github.com/QuivrHQ/quivr/issues/3667 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 2. security_permissions · 来源证据：[Bug]:
- Severity: high
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：[Bug]:
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源问题仍为 open，Pack Agent 需要复核是否仍影响当前版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_ff260ca3ab5247679b2ded201ec21e24 | https://github.com/QuivrHQ/quivr/issues/2004 | 来源讨论提到 docker 相关条件，需在安装/试用前复核。
## 3. installation · 来源证据：Integration idea: Screenpipe for screen/audio context
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：Integration idea: Screenpipe for screen/audio context
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_3a7f345691d04bbea72b03858746645e | https://github.com/QuivrHQ/quivr/issues/3658 | 来源类型 github_issue 暴露的待验证使用条件。
## 4. installation · 来源证据：[Bug]: RuntimeError: There is no current event loop in thread 'MainThread' when using Brain.from_files() in script
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个安装相关的待验证问题：[Bug]: RuntimeError: There is no current event loop in thread 'MainThread' when using Brain.from_files() in script
- User impact: 可能阻塞安装或首次运行。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_e3091b23b6184dffb982e0cc494134fd | https://github.com/QuivrHQ/quivr/issues/3650 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 5. capability · 能力判断依赖假设
- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: 假设不成立时，用户拿不到承诺的能力。
- Suggested check: 将假设转成下游验证清单。
- Guardrail action: 假设必须转成验证项；没有验证结果前不能写成事实。
- Evidence: capability.assumptions | github_repo:640079149 | https://github.com/QuivrHQ/quivr | README/documentation is current enough for a first validation pass.
## 6. runtime · 来源证据：The garbage collector is trying to clean up non-checked-in connection >, which will…
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_e6fdde799a2b4f53806121190979025a | https://github.com/QuivrHQ/quivr/issues/3654 | 来源讨论提到 python 相关条件，需在安装/试用前复核。
## 7. runtime · 来源证据：core: v0.0.25
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个运行相关的待验证问题：core: v0.0.25
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_7c6096fb5a6442c7b53068a8dd8e2c78 | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.25 | 来源类型 github_release 暴露的待验证使用条件。
## 8. runtime · 来源证据：core: v0.0.29
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个运行相关的待验证问题：core: v0.0.29
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_7eeeeb626a10476e820fcd651727a55a | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.29 | 来源讨论提到 node 相关条件，需在安装/试用前复核。
## 9. runtime · 来源证据：core: v0.0.33
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个运行相关的待验证问题：core: v0.0.33
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_d7a735cb41bc44f2abeb148d689f1320 | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.33 | 来源类型 github_release 暴露的待验证使用条件。
## 10. maintenance · 来源证据：core: v0.0.24
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：core: v0.0.24
- User impact: 可能影响升级、迁移或版本选择。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_5272fe70cda24220a5aeb994ad06819e | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.24 | 来源类型 github_release 暴露的待验证使用条件。
## 11. maintenance · 来源证据：core: v0.0.26
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个维护/版本相关的待验证问题：core: v0.0.26
- User impact: 可能增加新用户试用和生产接入成本。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_7f723b616fd446c3be43298d75fd6e8b | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.26 | 来源类型 github_release 暴露的待验证使用条件。
## 12. maintenance · 维护活跃度未知
- Severity: medium
- Evidence strength: source_linked
- Finding: 未记录 last_activity_observed。
- User impact: 新项目、停更项目和活跃项目会被混在一起，推荐信任度下降。
- Suggested check: 补 GitHub 最近 commit、release、issue/PR 响应信号。
- Guardrail action: 维护活跃度未知时，推荐强度不能标为高信任。
- Evidence: evidence.maintainer_signals | github_repo:640079149 | https://github.com/QuivrHQ/quivr | last_activity_observed missing
## 13. security_permissions · 下游验证发现风险项
- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: 下游已经要求复核，不能在页面中弱化。
- Suggested check: 进入安全/权限治理复核队列。
- Guardrail action: 下游风险存在时必须保持 review/recommendation 降级。
- Evidence: downstream_validation.risk_items | github_repo:640079149 | https://github.com/QuivrHQ/quivr | no_demo; severity=medium
## 14. security_permissions · 存在评分风险
- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: 风险会影响是否适合普通用户安装。
- Suggested check: 把风险写入边界卡，并确认是否需要人工复核。
- Guardrail action: 评分风险必须进入边界卡，不能只作为内部分数。
- Evidence: risks.scoring_risks | github_repo:640079149 | https://github.com/QuivrHQ/quivr | no_demo; severity=medium
## 15. security_permissions · 来源证据：core: v0.0.27
- Severity: medium
- Evidence strength: source_linked
- Finding: GitHub 社区证据显示该项目存在一个安全/权限相关的待验证问题：core: v0.0.27
- User impact: 可能影响授权、密钥配置或安全边界。
- Suggested check: 来源显示可能已有修复、规避或版本变化，说明书中必须标注适用版本。
- Guardrail action: 不得脱离来源链接放大为确定性结论；需要标注适用版本和复核状态。
- Evidence: community_evidence:github | cevd_027338ba02764da6b371970615591265 | https://github.com/QuivrHQ/quivr/releases/tag/core-0.0.27 | 来源类型 github_release 暴露的待验证使用条件。
## 16. maintenance · issue/PR 响应质量未知
- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: 用户无法判断遇到问题后是否有人维护。
- Suggested check: 抽样最近 issue/PR，判断是否长期无人处理。
- Guardrail action: issue/PR 响应未知时，必须提示维护风险。
- Evidence: evidence.maintainer_signals | github_repo:640079149 | https://github.com/QuivrHQ/quivr | issue_or_pr_quality=unknown
## 17. maintenance · 发布节奏不明确
- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: 安装命令和文档可能落后于代码，用户踩坑概率升高。
- Suggested check: 确认最近 release/tag 和 README 安装命令是否一致。
- Guardrail action: 发布节奏未知或过期时，安装说明必须标注可能漂移。
- Evidence: evidence.maintainer_signals | github_repo:640079149 | https://github.com/QuivrHQ/quivr | release_recency=unknown