Related Pages

Related topics: Router Implementations & Algorithms, Deployment, Configuration & Threshold Calibration

Overview & System Architecture

RouteLLM is an open-source framework for cost-effective LLM serving that automatically routes each user query to either a "strong" or a "weak" model based on predicted query difficulty. The framework is designed to reduce inference cost while preserving response quality by learning when a cheap model is sufficient and when a more capable model is required Source: README.md:1-40. At its core, RouteLLM acts as a thin, drop-in replacement layer that sits in front of two underlying OpenAI-compatible model endpoints.

Purpose and Scope

The primary goal of the project is to provide a practical, production-usable routing layer rather than just a research artifact. The codebase delivers:

• A Python SDK (routellm.controller.Controller) for embedding routing logic into existing applications.
• An OpenAI-compatible HTTP server (routellm.openai_server) that transparently forwards chat.completions requests to either of two upstream models.
• A pluggable router registry implementing multiple routing strategies such as matrix factorization (mf), BERT classifier (bert), random baseline (random), similarity-weighted, and causal LLM-based routing Source: routellm/controller.py:1-60.
• Pre-trained router checkpoints and evaluation utilities for benchmarking cost/quality trade-offs.

The scope is intentionally limited to two-model routing. Community discussions such as issue #23 highlight that multi-model routing (e.g., privacy- or domain-aware selection) is not yet supported out of the box Source: GitHub issue #23.

High-Level Architecture

The system is composed of three cooperating layers: a client-facing interface, a Controller / router selection layer, and the upstream model providers.

flowchart LR
    A[Client / SDK caller] --> B[OpenAI Server or Controller]
    B --> C{Router}
    C -->|route to strong| D[Strong Model Endpoint]
    C -->|route to weak| E[Weak Model Endpoint]
    D --> F[Response to Client]
    E --> F

The Client can be either an application using the Python SDK or any OpenAI-API-compatible client pointing at the bundled server. The Controller owns router instances and model clients; the Router decides per-query which upstream to invoke. Both the server and the SDK reuse the same Controller for consistent decision-making Source: routellm/controller.py:20-80.

Core Components

Controller

The Controller class is the central orchestrator. It is instantiated once with a router name, a strong model identifier, a weak model identifier, and optionally an OPENAI_API_BASE for each side Source: routellm/controller.py:30-70. Issue #27 documents a known limitation: the controller currently shares a single base URL environment variable, so running a strong and a weak model on different servers requires manual environment workarounds Source: GitHub issue #27. The controller exposes chat.completions.create(...) which mirrors the OpenAI client surface but intercepts the model parameter to perform routing.

Routers

Routers implement a small contract defined in routellm/routers/router.py, exposing acquire(<router_arg>) that returns a model name to forward the request to Source: routellm/routers/router.py:1-40. Implementations include:

• Matrix Factorization (mf) — projects both the query and the "strong vs. weak" latent factor into a shared space and scores their inner product. Training logic lives in routellm/routers/matrix_factorization/train_matrix_factorization.py. Issue #83 clarifies that the .npy file loaded during training is a precomputed, frozen query embedding cached to avoid recomputation Source: GitHub issue #83.
• BERT classifier (bert) — fine-tunes a small encoder to predict whether the strong model is required. Issue #67 confirms that the training script for this router is not yet shipped in the repository Source: GitHub issue #67.
• Random (random) — uniform baseline used for ablation studies.

Issue #84 reports that swapping the embedding model used by the mf router (e.g., to intfloat/e5-small-v2) currently fails because the cached .npy tensor and the configured model name are coupled Source: GitHub issue #84.

OpenAI-Compatible Server

routellm/openai_server.py launches a FastAPI server that mimics the /v1/chat/completions and /v1/models endpoints. It selects routers through the --routers CLI flag and reads model identifiers via --strong-model / --weak-model Source: routellm/openai_server.py:1-80. Issue #63 shows a common deployment failure where litellm.drop_params is required to drop unsupported fields when forwarding to certain upstream providers (e.g., Ollama chat models) Source: GitHub issue #63.

Package Surface and Dependencies

The top-level routellm/__init__.py re-exports Controller so callers can simply from routellm import Controller Source: routellm/__init__.py:1-20. Runtime dependencies are declared in pyproject.toml and include litellm (for talking to many backends), fastapi, uvicorn, transformers, and sentence-transformers Source: pyproject.toml:1-60.

Request Flow and Integration

A typical request proceeds as follows: the client issues a chat completion request; the server or SDK passes the message list to the controller; the controller calls router.acquire(...); the router returns strong or weak; the controller forwards the request to the corresponding model client via litellm.completion(...); the response is returned unchanged to the caller Source: routellm/controller.py:80-140. Because the surface mirrors OpenAI's, downstream tools — including LangChain via its ChatOpenAI adapter — integrate without modification, as confirmed by community discussion in issue #28 Source: GitHub issue #28.

Known Operational Limitations

Community issues surface several recurring operational pain points: rate-limit and authorization errors when using hosted strong models (issue #78), compatibility errors with litellm.drop_params (issue #63), and the inability to use distinct base URLs for each side of the controller (issue #27). Issue #82 also notes that full training scripts are not bundled for every router variant. Together these define the current boundaries of the architecture and inform the roadmap for multi-model routing, embedded training code, and richer configuration.

Related Pages

Related topics: Overview & System Architecture, Deployment, Configuration & Threshold Calibration, Extensibility, Training & Custom Integrations

Router Implementations & Algorithms

RouteLLM exposes a pluggable router abstraction whose sole responsibility is to decide, for a given user query, whether to forward the request to the strong model (e.g., GPT-4) or the weak model (e.g., Mixtral). All routers ultimately return a single binary win probability p_win against the strong model, which the controller then converts into a routing decision. Source: routellm/routers/routers.py

Router Registry and Common Interface

routellm/routers/routers.py acts as the central registry and dispatcher. It maps a string identifier (e.g., "mf", "sw", "bert", "causal_llm", "random") to a concrete router class and exposes a uniform get_router(router_name, **kwargs) factory. Source: routellm/routers/routers.py

Each router implements a consistent calculate_cost / calculate_win_rate style API so the OpenAI-compatible server (routellm.openai_server) can swap algorithms without changing upstream code. The registry is also the place where pre-trained artefacts (e.g., NumPy matrices, HuggingFace checkpoints) are loaded lazily on first use.

Matrix Factorization Router

The MF router learns latent embeddings for queries and for the two models, then scores a query by the dot product with the strong-model vector. Source: routellm/routers/matrix_factorization/model.py

The model is implemented as a thin PyTorch module holding two non-trainable buffers:

• query_embeddings – a pre-computed [N_queries, D] matrix loaded from a .npy artefact
• model_embeddings – a [2, D] matrix (one row per model) that *is* trained

Because query embeddings are fixed, training only updates model_embeddings via gradient descent on a Bradley–Terry-style loss. Source: routellm/routers/matrix_factorization/train_matrix_factorization.py

Community context: the meaning of the .npy file and the choice to keep query vectors frozen are recurring questions (see issue #83). Issue #84 reports friction when swapping the embedding model away from the default; the trainer currently expects a specific embedding dimensionality produced upstream.

Similarity-Weighted Router

The SW router avoids learned parameters and instead computes a *similarity-weighted* win-rate estimate over a pool of labelled calibration queries. Source: routellm/routers/similarity_weighted/utils.py

The runtime path is:

1. Encode the incoming query with the configured embedding model (generate_embeddings.py).
2. Compute cosine similarity between the query and every calibration example.
3. Convert similarities into weights and aggregate the binary strong-model labels into a single win probability.

Source: routellm/routers/similarity_weighted/generate_embeddings.py

This router is attractive for users who want a strong baseline without retraining, since the heavy artefact is just an embedding matrix plus labels.

Causal LLM Router

The causal-LM router reformulates routing as a next-token prediction task. Given the user prompt, a small language model (e.g., a fine-tuned Llama variant) produces a token whose logit is interpreted as the strong-model win probability. Source: routellm/routers/causal_llm/model.py

The module is responsible only for inference and token-to-probability decoding; the associated training pipeline lives outside the open-sourced repository, which mirrors the gap noted for the BERT router in issue #67.

Known Gaps Reflected in Community Discussions

• BERT training code missing – issue #67 asks for the BERT training script; it is not present, only an inference hook in the registry. The same gap applies to the causal_llm trainer.
• Custom embedding models for MF – issue #84 reports errors when overriding the embedding model; the MF trainer hard-codes the dimensionality expected from the upstream pipeline.
• Two-model assumption – issue #23 highlights that every router above is fundamentally binary (strong vs. weak). The registry has no multi-arm interface today.

flowchart LR
    Q[User Query] --> R{router.calculate_win_rate}
    R -->|p_win >= threshold| S[Strong Model]
    R -->|p_win < threshold| W[Weak Model]
    S --> Resp[Response]
    W --> Resp

For most users the practical recommendation is: start with mf or sw (training scripts are present), and treat bert / causal_llm as inference-only checkpoints unless the upstream training artefacts are obtained separately.

Related Pages

Related topics: Overview & System Architecture, Router Implementations & Algorithms

Deployment, Configuration & Threshold Calibration

Purpose and Scope

RouteLLM exposes three cooperating artefacts for bringing a router online: an OpenAI-compatible HTTP server for drop-in replacement of an OpenAI endpoint, a YAML configuration file that binds routers to model identifiers and endpoints, and a threshold-calibration tool that sets the operating point at which a query is sent to the strong model versus the weak model. Together they answer the operational questions *which model serves this request*, *which backend URL handles it*, and *how often the strong model is selected*. This page documents the deployment surfaces, the structure of config.example.yaml, and the calibration workflow that ties them together.

OpenAI-Compatible Server Deployment

The primary deployment surface is routellm/openai_server.py, launched as a Python module:

python -m routellm.openai_server \
  --routers mf \
  --weak-model ollama_chat/codeqwen \
  --strong-model openai/gpt-4o-mini

Source: routellm/openai_server.py:1-80. The server speaks the standard /v1/chat/completions protocol, so any OpenAI client — including LangChain configured with an OpenAI-compatible base_url — can target it without code changes (relevant to issue #28 on LangChain integration).

Launch flags:

For deployments in which the strong and weak models live behind different OpenAI-compatible URLs — for example two local Ollama servers on different ports, or one local model and one hosted model — the --config flag loads a YAML file specifying distinct api_base values. Source: config.example.yaml:1-40. This directly addresses the community use case of routing between two Ollama instances (issue #76) and the request for separate base URLs per model (issue #27).

An alternative is the in-process Python SDK path documented in examples/python_sdk.md, where the RouteLLM controller is instantiated inside application code and no HTTP server is involved. This is preferable when the client and router share a process and HTTP overhead is undesirable.

Configuration File Layout

config.example.yaml is the canonical template. Its top-level keys are:

• router: selects the router class (mf, bert, etc.) and its parameters — checkpoint path, embedding model identifier, hidden dimension.
• strong_model / weak_model: each entry holds a model_name (LiteLLM identifier) plus an optional api_base and api_key for non-default endpoints.
• threshold: a float (default 0.5) used by routers that emit a continuous win-rate score; queries scoring at or above the threshold are routed to the strong model.

Source: config.example.yaml:1-60. The router.embedding_model key is the configuration knob for swapping the MF router's encoder, which is the change requested in issue #84. A minimal working configuration is shown in the example below.

router:
  name: mf
  checkpoint_path: routellm/mf
  embedding_model: sentence-transformers/all-MiniLM-L6-v2

strong_model:
  model_name: openai/gpt-4o-mini
  api_base: https://api.openai.com/v1

weak_model:
  model_name: ollama_chat/llama3
  api_base: http://localhost:11434

threshold: 0.5

Source: examples/router_chat.py:1-40. The same structure is used by examples/routing_to_local_models.md when both endpoints point at local servers.

Threshold Calibration

Routers such as mf and bert emit a continuous score — the predicted probability that the strong model wins on a given query — which is then compared against the routing threshold. Because that threshold directly determines the cost/quality trade-off, it should be set empirically rather than left at its default.

routellm/calibrate_threshold.py sweeps candidate thresholds on a labelled evaluation set and reports the cost saving achieved at a fixed quality target (typically 95% of strong-model performance). Typical invocation:

python -m routellm.calibrate_threshold \
  --router mf \
  --config config.yaml \
  --target-quality 0.95

Source: routellm/calibrate_threshold.py:1-60. The script prints a recommended threshold, which is then written into config.example.yaml under threshold:. Re-calibration is recommended whenever the strong or weak model is swapped, since the predicted win-rate distribution shifts with model identity and a threshold tuned for one pair is suboptimal for another.

Known Deployment Pitfalls

Community-reported issues surface several recurring deployment failures:

• litellm.drop_params errors on the OpenAI server (issue #63) usually trace to a LiteLLM version mismatch; pinning the version recorded in the repository's dependency list resolves it.
• Authorization errors with the BERT router (issue #78) occur when the Hugging Face checkpoint download is blocked; pre-downloading weights or setting HF_TOKEN is required for restricted environments.
• Rate-limit errors on hosted models (issue #78) indicate that the strong-model endpoint is being hit too often — lowering the calibrated threshold shifts traffic back to the weak model.

Source: examples/routing_to_local_models.md:1-40.

Summary

Deployment in RouteLLM reduces to three artefacts: an OpenAI-compatible server for production traffic, a YAML configuration that binds routers to model endpoints and base URLs, and a calibration tool that sets the routing threshold. Keeping the strong/weak model identities, their api_base URLs, and the calibrated threshold in sync is the main operational responsibility, and re-calibration should accompany any change of model pair.

Related Pages

Related topics: Router Implementations & Algorithms, Deployment, Configuration & Threshold Calibration

Extensibility, Training & Custom Integrations

RouteLLM is designed as a modular routing framework where the core routing decision can be swapped, retrained, or replaced without rewriting the request pipeline. This page covers the extension points exposed by the router layer, the training workflows that ship in the repository, and the configuration hooks used to integrate custom models, embeddings, and external clients.

Router Abstraction and Registry

All routing strategies inherit from a common base class defined in the routers package. The registry pattern lets a single server process load one or more routers at startup and dispatch incoming requests to the active strategy.

• BaseRouter defines the contract that every strategy must implement: an acquire/acquire_batch method that maps a conversation to a (model, weight) tuple, plus a calculate_strong/calculate_weak pair that returns win-rate style probabilities.
• The ROUTERS dictionary in routellm/routers/routers.py maps short CLI aliases (mf, bert, random, causal_llm, similarity_weighted) to concrete classes, enabling runtime selection. Source: routellm/routers/routers.py:1-80
• Helper utilities for tokenization and message normalization are exposed through the same module so custom routers can reuse the prompt-construction logic instead of re-implementing it.

This registry is the primary extension point: to add a new router, you implement a subclass of BaseRouter, register it in ROUTERS, and reference the alias from the --routers CLI flag. Community issue #67 ("Can BERT router source code (esp. for training) be added to the repo?") and #82 ("Are training scripts available?") both reflect demand for visible training code for non-mf strategies, which is exactly what this registry pattern makes possible.

Training Matrix Factorization and Similarity-Weighted Routers

The repository ships end-to-end training scripts for two learned routers. They share a data-preparation step but diverge at the model-fitting stage.

Matrix Factorization (MF)

The script routellm/routers/matrix_factorization/train_matrix_factorization.py loads pre-computed query embeddings from a .npy file rather than encoding prompts at training time. This is intentional: the frozen embedding tensor (query_embedding) acts as a static lookup table of prompt features, while only the per-model weight vectors are updated by SGD. Community issue #83 ("What is the .npy file for query embedding in mf router training.") highlights that this file is the cached output of an upstream sentence-transformer run, which decouples the encoder from the router fit. Source: routellm/routers/matrix_factorization/train_matrix_factorization.py:58-62

The training loop reads a battle dataset (model A vs. model B win/loss labels), looks up the corresponding row in the .npy embedding matrix, and minimizes a binary cross-entropy loss against the model strength indicators. The resulting weight matrix is serialized to disk and reloaded by MatrixFactorizationRouter at inference time.

Similarity-Weighted

routellm/routers/similarity_weighted/generate_embeddings.py is a one-shot preprocessing utility: it encodes a calibration set of prompts with a sentence-transformer model and stores the vectors so the router can perform k-nearest-neighbor style strength estimation at request time. Because the embeddings are produced offline, the inference path stays constant-memory. Source: routellm/routers/similarity_weighted/generate_embeddings.py:1-50

Custom Encoders

Community issue #84 ("How can I use the different embedding model for mf router") asks whether alternatives like intfloat/e5-small-v2 can be substituted. The answer hinges on regenerating the .npy file with the new encoder before invoking the training script — the router itself is encoder-agnostic as long as the dimensionality matches the model's weight matrix. This is the supported path for plugging in a domain-specific encoder.

Controller Configuration and Custom Integrations

The Controller is the bridge between a routing decision and the downstream model client. It exposes hooks for plugging in different transport layers without touching the routers.

Source: routellm/controller.py:1-120

Community issue #27 ("Allow Separate Base URLs for Strong and Weak Models in Controller") is addressed exactly by the strong_model_url / weak_model_url parameters — they allow a user to point the strong side at OpenAI while keeping the weak side on a self-hosted Ollama or vLLM endpoint.

OpenAI-Compatible Server

routellm/openai_server.py exposes the same controller through an OpenAI-shaped HTTP endpoint, which is the integration surface most third-party tools (LangChain, LlamaIndex, raw openai-python clients) consume. A typical launch looks like:

python -m routellm.openai_server \
  --routers mf \
  --weak-model ollama_chat/llama3 \
  --config examples/config.example.yaml

The server reads examples/config.example.yaml to populate controller defaults, including model URLs and API keys. Community issue #76 ("Try to run 2 Ollama") demonstrates the same pattern with two local Ollama servers (Phi4 as strong, Llama3 8B as weak). Source: routellm/openai_server.py:1-80

LangChain and Other Clients

Issue #28 ("Langchain integration") asks how RouteLLM can be used with LangChain. Because the server is OpenAI-compatible, any client that accepts a base_url parameter can target it:

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    base_url="http://localhost:6060/v1",
    api_key="EMPTY",
    model="route",
)

No LangChain-specific adapter is required; the OpenAI surface is the integration boundary.

Limitations Reported by the Community

Several recurring friction points are documented in the issue tracker and are worth noting before extending RouteLLM:

• Two-model ceiling. Issue #23 requests support for more than two models. The current Controller and all shipped routers are built around a binary strong/weak decision; extending to N-way routing requires subclassing BaseRouter and adapting the controller's dispatch logic.
• BERT training code. Issue #67 notes that BERT router training code is not in the public repo. Practitioners who need it must train externally and load weights through a custom subclass.
• Embedding mismatches. Issue #84 shows that swapping the encoder fails silently unless the .npy is regenerated with matching dimensionality and the MF router's projection layer is re-initialized.
• Server-side litellm.drop_params errors. Issue #63 reports that certain model/provider pairs trigger drop_params errors in the server. This is a litellm compatibility issue, not a router issue, and is worked around by upgrading the litellm version pinned in pyproject.toml.

Summary of Extension Workflows

The framework's design keeps routing, training, and transport as independent layers, which is what makes the surface above stable even as new routers and providers are added.

Doramagic Pitfall Log

Found 11 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

1. Configuration risk: Configuration risk requires verification

• Severity: high
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/63

2. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/87

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/76

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/lm-sys/RouteLLM

5. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/84

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/lm-sys/RouteLLM

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/lm-sys/RouteLLM

9. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/78

10. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

11. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using RouteLLM with real data or production workflows.

• Add AgentWeb business data integration - github / github_issue
• litellm.drop_params error when running the openapi server - github / github_issue
• How can I use the different embedding model for mf router - github / github_issue
• What is the .npy file for query embedding in mf router training. - github / github_issue
• Can BERT router source code (esp. for training) be added to the repo? - github / github_issue
• Following demo it doesn't work - github / github_issue
• Try to run 2 Ollam - github / github_issue
• Capability evidence risk requires verification - GitHub / issue