Related Pages

Related topics: Core Architecture and Module Layout

Overview and Installation

RAGxplorer is an open-source Python package that helps developers and practitioners visualize and explore the internal behaviour of a Retrieval-Augmented Generation (RAG) pipeline. Rather than treating the retriever as a black box, RAGxplorer projects the embedded document chunks and the user query into a two-dimensional space using UMAP, so users can visually inspect which chunks the retriever considers most relevant. The project is distributed on PyPI under the name ragxplorer and is published alongside a Streamlit-based hosted web application for users who prefer not to install the library locally. Source: README.md:1-40

Purpose and Scope

The core purpose of RAGxplorer is to make retrieval quality observable. Given a PDF document and a query, the library loads and chunks the document, embeds each chunk through a user-selected embedding model, projects the resulting vectors to 2D, and highlights the chunks that would be returned by the configured retrieval strategy. This makes it easy to compare retrieval techniques — such as naive similarity, HYDE, and multi-query — on the same document and query. The package is intended as a diagnostic and educational tool for engineers tuning RAG systems, not as a production retrieval framework. Source: README.md:20-60

The scope is intentionally narrow: RAGxplorer focuses on visualization and inspection. It does not provide its own vector store, does not manage long-running document indexes, and does not replace frameworks such as LangChain or LlamaIndex. Instead, it composes with them and delegates embedding, chunking, and answer generation to those libraries. Pinecone support is tracked as an open enhancement request, indicating that the maintainers view RAGxplorer as a thin visualizer over multiple backends. Source: README.md:60-90 Community references: issues/38

Installation

RAGxplorer is installed from PyPI using the standard pip workflow:

pip install ragxplorer

The package metadata is defined in setup.py, which declares the project name, version, and the runtime dependencies required for embedding, retrieval, and projection. The most recent published release is v0.1.2, which packages the library for general PyPI distribution. Source: setup.py:1-40 Source: README.md:10-20

Because the package is a thin wrapper around LangChain, OpenAI, UMAP, and Plotly, all of these are pulled in transitively by the install command. A supplementary requirements.txt is provided for users who prefer an explicit, pinned dependency list — recommended for reproducible environments. Source: requirements.txt:1-20

⚠️ Known installation pitfall (Issue #53): The langchain dependency declared in setup.py is not pinned. Recent versions of LangChain removed the langchain.text_splitter module, so a fresh pip install ragxplorer can resolve to an incompatible LangChain release and fail at import time with ModuleNotFoundError: No module named 'langchain.text_splitter'. Until the dependency is pinned upstream, the recommended workaround is to pin LangChain manually before installing RAGxplorer:

```bash

pip install "langchain<0.2" ragxplorer

```

Source: setup.py:10-30 Community reference: issues/53

Dependencies and Runtime Requirements

RAGxplorer relies on a small set of external services and libraries at runtime:

Source: ragxplorer/__init__.py:1-40 Source: requirements.txt:1-20

For full functionality — including HYDE and multi-query retrieval — an OpenAI API key (or a compatible alternative such as Azure OpenAI or a local Ollama server) must be available in the environment. Source: README.md:30-60

Quick Start

The minimal end-to-end workflow is:

from ragxplorer import RAGxplorer

client = RAGxplorer(embedding_model="nomic-embed-text")
client.load_pdf(document_path="./tutorials/nais2023.pdf", verbose=True)
client.visualize_query("How will AI innovation be balanced with ethics?")

load_pdf ingests and chunks the document, while visualize_query embeds the query, projects both the chunks and the query into 2D, and renders an interactive Plotly figure with the most relevant chunks highlighted. Source: ragxplorer/__init__.py:20-80 Source: README.md:40-80

For users who prefer not to install the package, a hosted Streamlit application (app.py) exposes the same capabilities through a browser UI. Uploads to that webapp are processed transiently and are not persisted beyond the session. Community reference: issues/45, issues/44

Common First-Run Issues

Two recurring issues are worth highlighting before going further:

1. 3-D array shape error (Found array with dim 3. None expected <= 2.) — typically caused by an embedding model whose output shape is incompatible with UMAP's 2-D projection, or by a LangChain version mismatch. Verify the embedding model and LangChain version before opening an issue. Community reference: issues/46
2. Expected document to be a str when using retrieval_method="multi_qns" — the multi-query retriever expects string documents; ensure that chunking returns Document objects whose page_content is a string, especially when substituting Azure OpenAI for OpenAI. Community reference: issues/47

For development or contribution, the repository does not yet ship a test suite; adding one is tracked as an open enhancement. Community reference: issues/30

Related Pages

Related topics: Overview and Installation, Query Expansion and Retrieval Methods, Visualization Pipeline and Customization

Core Architecture and Module Layout

RAGxplorer is a lightweight Python library that helps developers and practitioners visually inspect how a Retrieval-Augmented Generation (RAG) pipeline retrieves context. Its core architecture follows a small, layered design where a single orchestrator class coordinates dedicated modules for retrieval, query expansion, and projection-based visualization.

High-Level Module Layout

The package is centered on a single user-facing class, RAGxplorer, exported from the top-level module. Internally it delegates work to three specialized submodules and a constants module:

Source: ragxplorer/ragxplorer.py:1-30 Source: ragxplorer/constants.py:1-40

This layout keeps each concern isolated: the orchestrator owns the user-facing workflow, while rag, query_expansion, and projections are independently testable units. The constants module avoids hard-coded prompts or model names from being scattered across files.

The `RAGxplorer` Orchestrator

RAGxplorer is the single entry point exposed to users via from ragxplorer import RAGxplorer. The class follows a typical three-step pipeline: load, retrieve, visualize.

from ragxplorer import RAGxplorer

client = RAGxplorer(embedding_model="nomic-embed-text")
client.load_pdf(document_path="./doc.pdf", verbose=True)
client.visualize_query("How will AI innovation be balanced with ethics?")

The constructor stores configuration (embedding model identifier, retrieval method, chunk size) and prepares an Ollama-compatible embedding endpoint. Source: ragxplorer/ragxplorer.py:1-60

Key methods:

• load_pdf — Uses LangChain's PyPDFLoader to ingest the document, splits the text into chunks, embeds each chunk through the configured model, and stores both raw chunks and embeddings on the instance.
• chunk_and_embed — Bridges the loader output and the internal vector store; this is where the chunks are turned into NumPy arrays suitable for projection.
• visualize_query — Accepts a natural-language question, embeds it, retrieves relevant chunks via the RAG module, projects both chunks and query onto a 2D plane, and renders an interactive plotly chart.

Because the orchestrator exposes only a few methods, integration code stays compact. Source: ragxplorer/ragxplorer.py:60-160

The `RAG` Module

rag.py encapsulates retrieval logic. It defines a RAG class that accepts a list of chunks, their embeddings, and a query, then returns the most relevant chunks. The class delegates query rewriting to query_expansion and prompt construction to constants, so its own surface area stays narrow. Source: ragxplorer/rag.py:1-80

Retrieval flow inside RAG:

1. Receive the user's query string and the configured retrieval method.
2. Call the appropriate expansion strategy from query_expansion.py.
3. Embed the (possibly expanded) query using the same embedding model.
4. Compute similarity (cosine or dot product) between the query embedding and all chunk embeddings.
5. Return the top-*k* chunks together with their similarity scores.

By isolating this logic, the orchestrator remains decoupled from retrieval mechanics, which is also helpful for future extensions such as Pinecone-backed vector stores (tracked as an open feature request in issue #38). Source: ragxplorer/rag.py:80-140

Query Expansion Strategies

query_expansion.py provides a small registry of strategies that rewrite a user's query before retrieval. The orchestrator selects the strategy based on the retrieval_method argument. The supported strategies, defined as constants in constants.py, are:

• naive — Pass the query through unchanged. Useful as a baseline.
• hyde — "Hypothetical Document Embeddings": asks the LLM to generate a synthetic answer, then embeds that answer instead of the raw question. Improves recall when the question is short or abstract.
• multi_qns — Generates several reformulations of the query and retrieves against the union of results, improving recall at the cost of extra LLM calls.

Source: ragxplorer/query_expansion.py:1-60 Source: ragxplorer/constants.py:20-80

The community has reported that the multi_qns path can raise Expected document to be a str when the backing LLM returns non-string outputs (issue #47). Because the expansion module is the sole producer of these LLM-derived strings, fixing the issue should be localized here — for example, by coercing LLM outputs to str before downstream use. Source: community context referencing issue #47.

Projection and Visualization

projections.py is responsible for turning high-dimensional embeddings into coordinates that can be plotted. It wraps UMAP and exposes a function (commonly project_embeddings) that returns 2D arrays. The orchestrator calls this function once for the chunk embeddings and again for the query embedding, then assembles a plotly figure distinguishing retrieved chunks from non-retrieved ones. Source: ragxplorer/projections.py:1-60

A common source of confusion reported by users (issue #46) is the error Found array with dim 3. None expected <= 2.. This originates here: if chunk embeddings have an extra dimension (for example, when the embedding model returns batched outputs that were not flattened), UMAP fails. The fix lives in the orchestration layer — chunks must be reshaped to 2D before being passed to the projection module.

End-to-End Data Flow

flowchart LR
    PDF[PDF Document] --> Loader[LangChain PyPDFLoader]
    Loader --> Chunker[Text Splitter]
    Chunker --> EmbedChunks[Embedding Model]
    EmbedChunks --> Store[(Chunk Embeddings)]
    Query[User Query] --> Expand[query_expansion.py]
    Expand --> EmbedQuery[Embedding Model]
    Store --> Rag[rag.py: similarity + top-k]
    EmbedQuery --> Rag
    Rag --> TopK[Top-k Chunks]
    Store --> Proj[projections.py: UMAP]
    TopK --> Proj
    Proj --> Plot[plotly Visualization]

Source: ragxplorer/ragxplorer.py:60-200 Source: ragxplorer/rag.py:1-140 Source: ragxplorer/projections.py:1-60

Known Architectural Constraints

A few constraints are worth noting for contributors:

• Pinned LangChain dependency: An unpinned langchain requirement has caused ModuleNotFoundError: No module named 'langchain.text_splitter' after a major LangChain release (issue #53). The orchestrator's reliance on LangChain's text_splitter import path means the setup.py constraint must stay aligned with the supported LangChain major version.
• Single dimensionality reduction technique: UMAP is currently the only projection method (issue #36). Adding t-SNE or PCA would require a new module analogous to projections.py rather than edits inside it, preserving the single-responsibility layout.
• No automated test suite: As of issue #30, the project lacks formal tests; the modular layout is, however, friendly to unit testing of rag.py, query_expansion.py, and projections.py in isolation.

Related Pages

Related topics: Core Architecture and Module Layout, Visualization Pipeline and Customization

Query Expansion and Retrieval Methods

Overview and Purpose

Query expansion is the mechanism by which RAGxplorer augments a user's raw question with additional context before embedding and retrieval. The library exposes three retrieval strategies — naive, HYDE, and multi_qns — selectable through the retrieval_method parameter on the RAGxplorer client. These strategies are implemented in ragxplorer/query_expansion.py and dispatched from ragxplorer/rag.py, while method names and prompt templates are centralized as constants in ragxplorer/constants.py.

The motivation for offering multiple retrieval methods is to help users compare how different expansion strategies affect which chunks surface as the most relevant neighbors in the 2D UMAP projection. Each method produces a different set of vectors to query against the chunk store, so the resulting visualization highlights different parts of the embedding space.

Source: ragxplorer/query_expansion.py:1-40 Source: ragxplorer/rag.py:1-60 Source: ragxplorer/constants.py:1-20

Supported Retrieval Methods

The three retrieval methods currently available are:

Each method is selected at the RAGxplorer client level and routed through a dispatcher in rag.py. The available keys are kept as string constants in constants.py so that downstream code and tutorials reference the same identifiers.

Source: ragxplorer/query_expansion.py:10-80 Source: ragxplorer/constants.py:5-25 Source: ragxplorer/rag.py:30-90

Implementation Flow

The end-to-end flow for a query visualization is:

1. load_pdf (defined in ragxplorer/loaders.py) reads the PDF, splits it into chunks using a LangChain text splitter, embeds each chunk via the configured embedding_model (Ollama by default, e.g. nomic-embed-text), and stores the chunk matrix in memory on the client.
2. visualize_query (in rag.py) accepts the raw question, looks up the chosen retrieval_method, and calls the corresponding function from query_expansion.py.
3. The expansion function invokes the LLM through the embedding/chat client, parses the response, and returns either a single string (for naive and HYDE) or a list of strings (for multi_qns).
4. The expanded text(s) are embedded, compared against the chunk embeddings using cosine similarity, and the top-k results are projected alongside the query vector with UMAP for plotting.

Source: ragxplorer/rag.py:40-130 Source: ragxplorer/query_expansion.py:20-100 Source: ragxplorer/embedding.py:1-60 Source: ragxplorer/loaders.py:1-50

Known Limitations and Community-Reported Issues

Several community-reported issues trace back to the query expansion pipeline:

• 3D array errors in Colab/Hugging Face Spaces: Users hit Found array with dim 3. None expected <= 2. because the multi_qns path returns a list of expanded queries, each producing its own 2D embedding, and the downstream UMAP reducer or scatter plot receives the stacked array without the expected reshape/np.mean step (issue #46).
• "Expected document to be a str" with multi_qns: When users swap the default OpenAI client for Azure OpenAI, the LLM response is sometimes returned as a list of message objects rather than a string. The expansion function then passes a non-string into the embedding step, which raises this error (issue #47). A workaround is to coerce the response to a plain str before embedding.
• Unpinned LangChain dependency: setup.py does not pin langchain, so pip install ragxplorer can pull a version that has removed langchain.text_splitter, which causes load_pdf to fail before the retrieval layer is even reached (issue #53).
• Single dimensionality reducer: Only UMAP is wired in. Requests to add t-SNE or PCA (issue #36) would require touching visualize.py rather than query_expansion.py, but the expansion methods are orthogonal to the choice of reducer.

Source: ragxplorer/query_expansion.py:50-120 Source: ragxplorer/rag.py:80-140 Source: setup.py:1-40

Extending With a New Method

To add a new retrieval method, three files typically need to change:

1. Append a new key and prompt template to ragxplorer/constants.py.
2. Implement the expansion function in ragxplorer/query_expansion.py and register it in the dispatcher used by rag.py.
3. Ensure the function returns either a str (single-query) or List[str] (multi-query), and that any list output is flattened/averaged before being fed to the embedding model, so that downstream UMAP and scatter code in ragxplorer/visualize.py receives a 2D array.

The __init__.py file re-exports the RAGxplorer class, so no changes there are required when adding internal methods.

Source: ragxplorer/constants.py:1-40 Source: ragxplorer/query_expansion.py:1-30 Source: ragxplorer/rag.py:1-50 Source: ragxplorer/__init__.py:1-20 Source: ragxplorer/visualize.py:1-80

Related Pages

Related topics: Core Architecture and Module Layout, Query Expansion and Retrieval Methods

Visualization Pipeline and Customization

Purpose and Scope

RAGxplorer's visualization pipeline is a thin orchestration layer that turns a source document into an interactive 2-D scatter plot of embedded chunks, with optional overlays that mark which chunks a retrieval strategy would surface for a given natural-language query. The pipeline is exposed both as a Python API (via the RAGxplorer class in ragxplorer/ragxplorer.py) and as a Streamlit front-end (via app.py), so the same stages execute in both surfaces.

The pipeline is intentionally minimal: there are exactly four user-visible stages — load_pdf, chunk, embed, project — followed by the query-time visualize_query step. Source: ragxplorer/ragxplorer.py:1-40. This bounded design is what makes the pipeline easy to customize: each stage exposes a small set of knobs rather than a deep class hierarchy.

Pipeline Stages

chunk delegates to LangChain's RecursiveCharacterTextSplitter under the hood, which is why an unpinned LangChain dependency has caused installation failures in the wild — langchain.text_splitter was removed in newer releases. Source: setup.py:1-30, issue #53.

project always returns a 2-D NumPy array of shape (n_chunks, 2). This contract is the source of the well-known Found array with dim 3. None expected <= 2 error reported in issue #46: any embedding model that returns vectors with an extra dimension (for example, some HF pipelines that wrap outputs in a batch axis) will be rejected by UMAP. Source: ragxplorer/projections.py:1-40, issue #46.

flowchart LR
  A[load_pdf] --> B[chunk]
  B --> C[embed]
  C --> D[project / UMAP]
  D --> E[visualize_query]
  Q[User Query] --> E
  E --> R[Plotly scatter]

Customization Points

Embedding model selection. RAGxplorer(embedding_model=...) accepts any name that the local Ollama-compatible endpoint can resolve. There is no built-in fallback for OpenAI, Cohere, or Hugging Face endpoints, which is why users have forked the client to swap in AzureOpenAI (issue #47). When swapping providers, callers must guarantee that the returned vector is strictly (n, d) — any wrapper batch dimension will trip UMAP. Source: ragxplorer/ragxplorer.py:1-80, issue #47.

Retrieval method. visualize_query accepts retrieval_method with supported values naive, hyde, and multi_qns. Each strategy produces a different ranking over the embedded chunks, and the visualization colors the top-k hits to make the strategy's bias visible. multi_qns expands the user query into several sub-questions via the same LLM endpoint; users have reported that a non-string intermediate (e.g., a dict) returned by a custom LLM wrapper surfaces as Expected document to be a str (issue #47). Source: ragxplorer/ragxplorer.py:80-160, issue #47.

Visual styling. visualize_query exposes color, point_size, and point_alpha parameters that feed directly into the underlying plotly.express.scatter call. The Plotly figure is returned to the caller rather than rendered inline, which lets both the Streamlit app and standalone notebooks render it through st.plotly_chart or fig.show() respectively. Source: app.py:1-80, ragxplorer/ragxplorer.py:80-160.

Dimensionality reduction. UMAP is the only projection backend shipped today. Community request #36 explicitly asks for t-SNE and PCA back-ends, and the existing projections.py module is the natural extension point — it currently exposes a single function that wraps umap.UMAP(n_components=2) and is called unconditionally from project(). Source: ragxplorer/projections.py:1-40, issue #36.

Known Limitations and Community-Reported Friction

Three recurring classes of failure show up in the issue tracker and are worth knowing before customizing the pipeline:

1. Embedding dimensionality. UMAP refuses any vector with rank ≠ 2; if your embedding endpoint returns (batch, n, d) or (n, 1, d), you must squeeze before calling embed(). Source: issue #46.
2. LangChain API drift. The package imports langchain.text_splitter, which was relocated in LangChain 1.x. Pinning or migrating the import is required for fresh installs. Source: issue #53.
3. Single projection algorithm. Adding method="tsne" or method="pca" to project() is the most requested extension and would not require touching the rest of the pipeline, since every downstream consumer expects only (n, 2) output. Source: issue #36.

The Streamlit surface in app.py does not introduce any additional pipeline stage — it simply calls the same Python API and renders the returned Plotly figure through st.plotly_chart. Privacy questions about uploaded PDFs (issues #44, #45) are scoped to the hosted Streamlit deployment and do not affect the local Python pipeline, which never persists the document beyond the in-memory chunks list on the client instance. Source: app.py:1-120, issue #45.

Doramagic Pitfall Log

Found 11 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/gabrielchua/RAGxplorer/issues/3

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/gabrielchua/RAGxplorer/issues/41

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/gabrielchua/RAGxplorer/issues/47

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/gabrielchua/RAGxplorer

5. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gabrielchua/RAGxplorer

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/gabrielchua/RAGxplorer

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/gabrielchua/RAGxplorer

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/gabrielchua/RAGxplorer/issues/29

9. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/gabrielchua/RAGxplorer/issues/53

10. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gabrielchua/RAGxplorer

11. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/gabrielchua/RAGxplorer

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using RAGxplorer with real data or production workflows.

• Unpinned dependency in setup.py causes latest LangChain install, causing - github / github_issue
• Found array with dim 3. None expected <= 2. - github / github_issue
• When choose retrieval_method= "multi_qns" get error: "Expected document - github / github_issue
• Write Tests - github / github_issue
• Data Privacy - github / github_issue
• Sweep: RAGxplorer Demo link in README not working - github / github_issue
• Data Privacy - github / github_issue
• Pinecone support - github / github_issue
• Feedback and Suggestions to Improve this Project - github / github_issue
• Create tutorials - github / github_issue
• v0.1.2 - github / github_release
• Capability evidence risk requires verification - GitHub / issue