Related Pages

Related topics: Architecture and Core Components, Supported Models and Customization

Overview and Getting Started

Purpose and Scope

FlashRank is a lightweight, ultra-fast Python library that adds neural re-ranking capabilities to existing search and retrieval pipelines. It is designed as a drop-in post-retrieval refinement step: given a query and a list of candidate passages produced by a first-stage retriever (BM25, dense embeddings, hybrid search, etc.), FlashRank re-scores and re-orders those passages using a more powerful cross-encoder, LLM listwise, or RankT5 model. The library handles model download, caching, tokenization, ONNX inference, and result formatting, exposing a minimal API surface so that integrators do not need to understand the internals of the underlying reranker.

The scope of the project deliberately excludes the retrieval stage itself. FlashRank does not index documents, build vector stores, or fetch candidates from a search backend. Its single responsibility is to take query + passages → ranked passages. The library targets two primary use cases: (1) RAG pipelines where a retriever returns the top-k chunks and a reranker is needed to improve precision, and (2) standalone re-ranking evaluation workflows where researchers want to compare reranker models on a fixed candidate set.

Source: README.md:1-40

Installation

Installation is published as a standard PyPI package. The recommended install pulls in the core ONNX runtime, tokenizer utilities, and HTTP client used to fetch model artifacts from the Hugging Face Hub.

pip install flashrank

The setup.py and pyproject.toml declare the runtime dependencies (numpy, torch/onnxruntime, tokenizers, tqdm, requests) and the version metadata. There is no mandatory GPU dependency: the default inference path runs on CPU via ONNX, which is consistent with the project's positioning as a lightweight reranker. A community request for optional GPU/CUDA acceleration is tracked but not yet the default (Issue #8).

Source: setup.py:1-60 Source: pyproject.toml:1-40

Core API and Minimal Workflow

The public API is intentionally small. After installation, the typical workflow is three steps: instantiate a Ranker, call ranker.rerank(passages), and consume the sorted result.

from flashrank import Ranker, RerankRequest

# Step 1: load a supported reranker (downloads/caches the model on first use)
ranker = Ranker(model_name="ms-marco-TinyBERT-L-2-v2")

# Step 2: build a rerank request
query = "How to speed up python list comprehension?"
passages = [
    {"id": 1, "text": "Use map() instead of loops for simple transforms..."},
    {"id": 2, "text": "List comprehensions are already optimized in CPython..."},
    {"id": 3, "text": "How to configure NGINX for high concurrency..."},
]

request = RerankRequest(query=query, passages=passages)

# Step 3: rerank — returns the same passages sorted by relevance score
results = ranker.rerank(request)
for r in results:
    print(r["id"], r["score"], r["text"])

The Ranker class is the central entry point defined in flashrank/Ranker.py. It owns the model directory, the tokenizer, and the inference session. Each call to rerank() performs tokenization for every (query, passage) pair, runs a single batched forward pass, and assigns a continuous score used to sort the candidates. A useful side effect of the design is that the id field on each passage is preserved in the output, which lets callers map the reranked list back to their original metadata — a behavior explicitly requested by the community in Issue #2.

Source: flashrank/__init__.py:1-20 Source: flashrank/Ranker.py:1-80 Source: examples/basic_example.py:1-30

Supported Models and Configuration

FlashRank ships with a registry of pre-converted ONNX rerankers. The registry is resolved at Ranker construction time using the model_name parameter, and the model archive is fetched from a dedicated Hugging Face organization (prithivida/flashrank). The supported families include:

The mapping from model_name to the Hugging Face archive URL, default cache directory, max sequence length, and score normalization behavior is centralized in flashrank/config.py. The Ranker constructor accepts overrides such as cache_dir for fully local model resolution — a workflow the community has been actively asking about, since some deployments (air-gapped, restricted egress) cannot reach huggingface.co (Issue #40 about SSL errors in Docker, Issue #37 about local model support).

Source: flashrank/config.py:1-120 Source: README.md:40-120

Common Pitfalls and Operational Notes

Three recurring failure modes show up in community reports and are worth noting for new users:

1. SSL / network errors on first run. The first call to Ranker(...) triggers a download from the Hugging Face Hub. In hardened Docker images or behind certain proxies, this fails with SSLError: UNSAFE_LEGACY_RENEGOTIATION_DISABLED. The fix is either to update the OpenSSL/requests stack in the container or to pre-download the model archive and point cache_dir at it.
2. ResourceWarning on unclosed file descriptors. A previous version of Ranker.py opened special_tokens_map.json without an explicit close(), leading to ResourceWarning under long-lived processes. The issue was fixed upstream and shipped in the 0.2.9 / 0.2.10 release line (Issue #25, PR #38).
3. Original index / metadata mapping. Because rerank() returns sorted passages, the id field is the only reliable handle back to the source metadata. Always pass stable identifiers into the input passages.

Once these are understood, FlashRank integrates cleanly into any retrieval pipeline as a single function call between the first-stage retriever and the downstream LLM or UI.

Related Pages

Related topics: Overview and Getting Started, Supported Models and Customization, Deployment, Performance, and Troubleshooting

Architecture and Core Components

Overview

FlashRank is a lightweight, ONNX-based reranking library designed to act as a second-pass reranker on top of first-stage retrievers. The architecture follows a deliberately small two-layer model: a static configuration layer that enumerates supported pretrained rerankers, and a runtime Ranker class that downloads, caches, and executes the chosen model. The design intentionally avoids GPU dependencies (see issue #8) to keep inference cheap and deployable in CPU-only environments such as Docker containers, where SSL-only failures have been reported (issue #40).

The package is shipped to PyPI (releases such as 0.2.4 adding LLM-based listwise rerankers, 0.2.9 for minor fixes, and 0.2.10 for the file-descriptor fix) via setup.py, and exposes only a tiny public surface from flashrank/__init__.py.

Core Components

Package Entry Point and Public API

flashrank/__init__.py exposes the public surface — typically the Ranker class and a small set of model-name constants — so that users can write from flashrank import Ranker without needing to touch submodules. The package keeps its export list intentionally short to make breaking changes easier to reason about. Source: flashrank/__init__.py:1-20; setup.py:1-60.

Configuration Layer

flashrank/Config.py holds a static registry (commonly named default_model_config) that maps short, human-friendly model identifiers — for example ms-marco-TinyBERT-L-2-v2, rank-T5-flan (added in 0.1.64), and the listwise LLM-based rerankers added in 0.2.4 — to metadata such as the Hugging Face repo, the ONNX file name, and the model family. Each entry points to a remote artifact under huggingface.co/prithivida/flashrank/resolve/main/...zip. Source: flashrank/Config.py:1-80.

This indirection is what allows the runtime to remain model-agnostic: a single string passed to Ranker(...) resolves through the config table to a complete download/cache/recipe description, without any conditional logic inside the inference path.

The `Ranker` Class

flashrank/Ranker.py contains the Ranker class, which is the heart of the library. Its responsibilities are:

1. Model resolution — take a model_name, look it up in Config.py, and materialize a local path (downloading the zip from Hugging Face on first use, then reusing the cache on subsequent calls).
2. Tokenizer + ONNX session bootstrap — load special_tokens_map.json and the ONNX session with the providers appropriate for the host. A historical bug used json.load(open(...)) without a context manager, which produced ResourceWarning and unclosed-file-descriptor issues (issue #25). The fix landed in PR #38 and shipped in 0.2.10.
3. State retention — the constructor stores the input list of passages as an instance attribute. This pattern was highlighted in issue #45 as having surprising memory implications when callers reuse the same Ranker instance across large batches, because the original list is held for the lifetime of the object.
4. rerank entry point — the rerank(passages, query, top_n=...) method tokenizes each (query, passage) pair, runs a single forward pass through the ONNX graph, and returns the input passages decorated with a score field, sorted in descending order.

Source: flashrank/Ranker.py:1-120, specifically the construction path at line 25 referenced in issue #45.

Parser and Utilities

flashrank/Parser.py provides a small helper that normalizes user-supplied passages into the {"id", "text", "meta"} schema expected by Ranker.rerank. The companion flashrank/utils.py hosts generic helpers such as cache-directory resolution and zip extraction. The package keeps these in separate modules so that the hot path inside Ranker.rerank remains lean and free of I/O concerns. Source: flashrank/Parser.py:1-60; flashrank/utils.py:1-60.

End-to-End Reranking Flow

flowchart LR
    A[User code] --> B["Ranker.__init__(model_name)"]
    B --> C{Config.py lookup}
    C -->|hit| D[Cache directory check]
    C -->|miss| E[HTTP GET to HF zip]
    E --> F[Unzip into cache]
    D --> G[Load tokenizer + ONNX session]
    F --> G
    G --> H["Ranker.rerank(passages, query)"]
    H --> I[ONNX forward pass per pair]
    I --> J[Sorted passages with score + meta]

The user instantiates a Ranker once with a model name from the Config.py registry; FlashRank handles download, caching, and session creation, then reuses that session for every subsequent rerank call. Because the original passages are held on the instance (issue #45), the recommended pattern is to scope one Ranker per request lifecycle or clear state explicitly between large batches.

Known Architectural Constraints

Several limitations surface repeatedly in community discussions and are baked into the current design:

• CPU-only inference. No CUDA / multi-GPU path exists yet (issue #8, still open as of the latest release).
• Registered models only. Custom local rerankers are not first-class; users must either add an entry to Config.py or place an ONNX export locally and dig into Ranker.py (issue #37).
• ONNX-only format. Authors must convert their HF model to ONNX before use; the maintainers do not ship a generic converter (issue #41).
• Hugging Face network dependency. The first call requires outbound HTTPS to huggingface.co, which can fail in restricted networks — for example with UNSAFE_LEGACY_RENEGOTIATION_DISABLED errors on older OpenSSL builds inside Docker (issue #40).
• Original index preservation. There is no separate original_index field; the supported way to round-trip back to upstream metadata is the per-passage meta dictionary (issue #2).

Source: README.md:1-200; flashrank/Config.py:1-80; flashrank/Ranker.py:1-120.

Related Pages

Related topics: Overview and Getting Started, Architecture and Core Components, Deployment, Performance, and Troubleshooting

Supported Models and Customization

FlashRank ships with a small registry of pre-built ONNX reranker models, but the Ranker class also accepts a local ONNX model directory for fully custom use. This page documents the registered models, the Ranker configuration surface, and the supported customization paths (local model directory, cache directory, ranking mode, max length), while also flagging community-reported limitations around new architectures, GPU execution, and network access.

Registered Models Registry

The default set of supported models lives in flashrank/Config.py. Each entry maps a friendly model_name string to a Hugging Face download URL, the expected ONNX file name inside the archive, and a default max_length. The registry is the single source of truth for the "registered models" used in tutorials.

Source: flashrank/Config.py:1-120

Typical families represented in the registry include:

If a desired cross-encoder is not in this registry (for example cross-encoder/ms-marco-MiniLM-L-6-v2 raised in issue #26), it must be loaded via the local-model path described below rather than by name.

Source: README.md:40-90

Ranker Configuration Surface

The user-facing entry point is the Ranker class defined in flashrank/Ranker.py. Its constructor accepts the following parameters, which together define the supported customization surface:

• model_name — either a key from the Config.py registry, or a free-form identifier used only for logging when a local model is supplied.
• model_dir — local directory containing a pre-converted ONNX model plus its tokenizer files. When provided, FlashRank skips downloading from Hugging Face.
• cache_dir — overrides the default location for downloaded registered models.
• max_length — caps the tokenized sequence length; defaults to the value declared in Config.py for the chosen model.
• ranking_mode — selects the reranking strategy (e.g., cross-encoder for score-based ranking, or listwise modes such as monot5 / rankpair).

Source: flashrank/Ranker.py:1-60

These parameters are the contract exposed to callers; everything else in the class (self.passages, internal session objects, etc.) is implementation detail and should not be relied on (see issue #45 about hidden state attached to the instance).

Using a Local ONNX Model

Issue #37 asks how to use a custom local reranker. The supported path is:

1. Convert the cross-encoder to ONNX externally (issue #41 discusses this workflow for models such as mixedbread-ai/mxbai-rerank-base-v2, which FlashRank does not plan to bundle).
2. Place the .onnx file together with its tokenizer assets (tokenizer.json, special_tokens_map.json, config.json, etc.) into a single directory.
3. Instantiate the ranker with Ranker(model_name="my-custom", model_dir="/path/to/dir").

Source: flashrank/Ranker.py:20-55

Because downloads from huggingface.co can fail in restricted environments (issue #40 reports an SSLError inside Docker), shipping a local model is also the recommended workaround for offline or air-gapped deployments. The cache_dir argument is the related escape hatch when the network is reachable but the default cache path is not writable.

Known Limitations and Community Workarounds

Several features requested by the community are not supported by the customization surface described above:

• GPU / CUDA execution. Issue #8 requests optional GPU inference. The current Ranker only drives an ONNX Runtime session on CPU, so custom models that need acceleration must rely on ONNX Runtime execution providers configured outside of FlashRank.
• File descriptor leaks. Before v0.2.10, helper functions in the model-loading path opened JSON files without with blocks, producing ResourceWarning messages (issue #25). Local users who upgrade should ensure they are on v0.2.10 or later to inherit the fix in flashrank/Ranker.py.
• Architecture compatibility. Listwise rerankers and RankT5 expect a different output shape than cross-encoders; a locally converted model that is not on the registered list must match one of the supported ranking_mode values to be scored correctly.
• PyPI distribution of new releases. Issue #39 tracks the gap between merged fixes and published wheels — the customization behavior documented here corresponds to the latest released version, not necessarily the main branch.

Source: flashrank/Ranker.py:25-45, README.md:1-40, setup.py:1-40, pyproject.toml:1-40

In short, customization in FlashRank is intentionally narrow: pick a registered name, point at a local ONNX directory, or override the cache. Anything beyond that — new architectures, GPU, or richer ranking modes — is a community-acknowledged gap rather than an unsupported configuration flag.

Related Pages

Related topics: Overview and Getting Started, Architecture and Core Components, Supported Models and Customization

Deployment, Performance, and Troubleshooting

FlashRank is a lightweight Python library that augments existing retrieval pipelines with neural re-ranking using ONNX-converted transformer models. Operating concerns fall into three broad categories: how the package is shipped and run in production environments, how the inference path performs under real workloads, and how to diagnose the most frequently reported runtime errors. This page consolidates guidance for each of these areas based on the repository's source code and the most active community threads.

Deployment Patterns

The package is published to PyPI and is installed as a standard Python dependency. The setup.py declares onnxruntime and other runtime requirements, while flashrank/Config.py defines a default_model_id and a default_cache_dir that control where downloaded artifacts are stored. Source: setup.py:1-80

The first call to Ranker(...) triggers a download of the model archive from huggingface.co/prithivida/flashrank/... into a local cache directory. The archive is then extracted. This lazy, network-driven initialization has two deployment consequences:

• Containers must allow outbound HTTPS to HuggingFace. Community issue #40 reports SSLError: UNSAFE_LEGACY_RENEGOTIATION_DISABLED when running inside a hardened Docker image. The cause is a stale OpenSSL in the base image rather than a FlashRank defect; rebuilding the image with a current OpenSSL or pinning a non-legacy TLS profile resolves it.
• Air-gapped or offline clusters can pre-seed the cache. Issue #37 shows that pointing cache_dir to a pre-populated folder containing the unzipped model (config.json, tokenizer.json, model.onnx, special_tokens_map.json) lets Ranker load locally without contacting HuggingFace. Source: flashrank/Ranker.py:1-50

Performance Characteristics

FlashRank targets CPU-first, low-latency re-ranking via ONNX Runtime. The inference loop is implemented in Ranker.rerank(), which tokenizes each query/passage pair, runs the ONNX session synchronously, and applies a sigmoid to the raw logit. Source: flashrank/Ranker.py:20-90

Key performance considerations drawn from the source and community reports:

• Single-threaded per call. The ranker instance stores the original passage list on self (issue #45), so concurrent calls from different threads share the same attribute write path. In practice, deploy FlashRank as a stateless worker and call it from a thread pool sized to available CPU cores.
• GPU/CUDA is not first-class. Issue #8 is the long-standing request to expose providers=["CUDAExecutionProvider"] to the underlying ONNX session. As of the current Ranker.py, the ONNX session is constructed with default providers, meaning inference is CPU-bound. Users who need GPU must patch the session creation or contribute a device argument.
• Listwise/LLM rerankers are heavier. Release 0.2.4 introduced LLM-based listwise rerankers; they trade latency for accuracy and should be treated as a separate tier from the default cross-encoder ONNX models listed in Config.py. Source: README.md:1-120

Common Issues and Troubleshooting

The most common runtime problems map to a small number of well-known root causes.

1. ResourceWarning about unclosed files. Issue #25 and PR #38 addressed the use of bare open(...) calls in the tokenizer warm-up path. The fix landed in the 0.2.9/0.2.10 line and wraps the file handle in a with block. Users still on older pinned versions should upgrade. Source: flashrank/Ranker.py:1-60

2. SSL failures behind corporate proxies or in slim Docker images. As described in issue #40, the failure is environmental. Either rebuild the image with a current openssl / ca-certificates, set HF_HUB_DISABLE_SSL_VERIFICATION (not recommended for production), or pre-seed the cache as described above.

3. Model not in the registered list. Config.py enumerates a finite set of supported model IDs. Issue #26 requests adding ms-marco-MiniLM-L-6-v2, and issue #41 asks how to ONNX-convert arbitrary models. The supported path is: convert the model with optimum-cli export onnx, zip the artifacts, host them where Ranker can reach, and load via the model_path argument. Source: flashrank/Config.py:1-60

4. Original index lost after re-ranking. Issue #2 notes that the returned list does not carry the source position. Because rerank() assigns id values based on the input order, callers can preserve a mapping {id: original_index} before invoking rerank() to re-attach metadata.

5. Memory growth in long-lived processes. Issue #45 highlights that self.passages is retained on the instance. For long-running services, instantiate a fresh Ranker per worker or explicitly delete the attribute between batches to release references to large passage objects.

Operational Checklist

A minimal production checklist derived from the above:

1. Pin a FlashRank version >= 0.2.10 to inherit the file-handle fix.
2. Pre-seed cache_dir in your container image to avoid first-call network latency and to support air-gapped deploys.
3. Run FlashRank as a CPU-bound, stateless service sized to the number of physical cores.
4. If you need GPU, plan to fork or patch Ranker.py until upstream exposes a device parameter.
5. Track upstream issues #8, #40, #41, and #45 for known operational gaps.

With these patterns, FlashRank integrates predictably into retrieval stacks while its small, well-defined failure modes remain easy to diagnose.

Doramagic Pitfall Log

Found 10 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/PrithivirajDamodaran/FlashRank/issues/40

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/PrithivirajDamodaran/FlashRank/issues/39

3. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/PrithivirajDamodaran/FlashRank

4. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/PrithivirajDamodaran/FlashRank/issues/41

5. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/PrithivirajDamodaran/FlashRank

6. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/PrithivirajDamodaran/FlashRank

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/PrithivirajDamodaran/FlashRank

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/PrithivirajDamodaran/FlashRank/issues/25

9. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/PrithivirajDamodaran/FlashRank

10. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/PrithivirajDamodaran/FlashRank

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using FlashRank with real data or production workflows.

• hugging face ssl error - github / github_issue
• local reranker model support? - github / github_issue
• onnx conversion - github / github_issue
• Request: release v0.2.10 - github / github_issue
• ResourceWarning: Enable tracemalloc to get the object allocation traceba - github / github_issue
• Option to Use GPU, CUDA - github / github_issue
• Minor fixes - github / github_release
• Stable release with LLM based Listwise Rerankers - github / github_release
• Stable release with Rank T5 Support - github / github_release
• Capability evidence risk requires verification - GitHub / issue