# https://github.com/lm-sys/RouteLLM Project Manual

Generated at: 2026-07-24 20:54:20 UTC

## Table of Contents

- [Overview & System Architecture](#page-overview)
- [Router Implementations & Algorithms](#page-routers)
- [Deployment, Configuration & Threshold Calibration](#page-deployment)
- [Extensibility, Training & Custom Integrations](#page-extensibility)

<a id='page-overview'></a>

## Overview & System Architecture

### Related Pages

Related topics: [Router Implementations & Algorithms](#page-routers), [Deployment, Configuration & Threshold Calibration](#page-deployment)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [routellm/__init__.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/__init__.py)
- [routellm/controller.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/controller.py)
- [routellm/openai_server.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/openai_server.py)
- [routellm/routers/router.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/router.py)
- [routellm/routers/matrix_factorization/router.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/matrix_factorization/router.py)
- [routellm/routers/bert/router.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/bert/router.py)
- [README.md](https://github.com/lm-sys/RouteLLM/blob/main/README.md)
- [pyproject.toml](https://github.com/lm-sys/RouteLLM/blob/main/pyproject.toml)
</details>

# Overview & System Architecture

RouteLLM is an open-source framework for cost-effective LLM serving that automatically routes each user query to either a "strong" or a "weak" model based on predicted query difficulty. The framework is designed to reduce inference cost while preserving response quality by learning when a cheap model is sufficient and when a more capable model is required `Source: [README.md:1-40]()`. At its core, RouteLLM acts as a thin, drop-in replacement layer that sits in front of two underlying OpenAI-compatible model endpoints.

## Purpose and Scope

The primary goal of the project is to provide a practical, production-usable routing layer rather than just a research artifact. The codebase delivers:

- A Python SDK (`routellm.controller.Controller`) for embedding routing logic into existing applications.
- An OpenAI-compatible HTTP server (`routellm.openai_server`) that transparently forwards `chat.completions` requests to either of two upstream models.
- A pluggable router registry implementing multiple routing strategies such as matrix factorization (`mf`), BERT classifier (`bert`), random baseline (`random`), similarity-weighted, and causal LLM-based routing `Source: [routellm/controller.py:1-60]()`.
- Pre-trained router checkpoints and evaluation utilities for benchmarking cost/quality trade-offs.

The scope is intentionally limited to **two-model routing**. Community discussions such as issue #23 highlight that multi-model routing (e.g., privacy- or domain-aware selection) is not yet supported out of the box `Source: GitHub issue #23`.

## High-Level Architecture

The system is composed of three cooperating layers: a **client-facing interface**, a **Controller / router selection layer**, and the **upstream model providers**.

```mermaid
flowchart LR
    A[Client / SDK caller] --> B[OpenAI Server or Controller]
    B --> C{Router}
    C -->|route to strong| D[Strong Model Endpoint]
    C -->|route to weak| E[Weak Model Endpoint]
    D --> F[Response to Client]
    E --> F
```

The **Client** can be either an application using the Python SDK or any OpenAI-API-compatible client pointing at the bundled server. The **Controller** owns router instances and model clients; the **Router** decides per-query which upstream to invoke. Both the server and the SDK reuse the same `Controller` for consistent decision-making `Source: [routellm/controller.py:20-80]()`.

## Core Components

### Controller

The `Controller` class is the central orchestrator. It is instantiated once with a router name, a strong model identifier, a weak model identifier, and optionally an `OPENAI_API_BASE` for each side `Source: [routellm/controller.py:30-70]()`. Issue #27 documents a known limitation: the controller currently shares a single base URL environment variable, so running a strong and a weak model on different servers requires manual environment workarounds `Source: GitHub issue #27`. The controller exposes `chat.completions.create(...)` which mirrors the OpenAI client surface but intercepts the `model` parameter to perform routing.

### Routers

Routers implement a small contract defined in `routellm/routers/router.py`, exposing `acquire(<router_arg>)` that returns a model name to forward the request to `Source: [routellm/routers/router.py:1-40]()`. Implementations include:

- **Matrix Factorization (`mf`)** — projects both the query and the "strong vs. weak" latent factor into a shared space and scores their inner product. Training logic lives in `routellm/routers/matrix_factorization/train_matrix_factorization.py`. Issue #83 clarifies that the `.npy` file loaded during training is a precomputed, frozen query embedding cached to avoid recomputation `Source: GitHub issue #83`.
- **BERT classifier (`bert`)** — fine-tunes a small encoder to predict whether the strong model is required. Issue #67 confirms that the training script for this router is not yet shipped in the repository `Source: GitHub issue #67`.
- **Random (`random`)** — uniform baseline used for ablation studies.

Issue #84 reports that swapping the embedding model used by the `mf` router (e.g., to `intfloat/e5-small-v2`) currently fails because the cached `.npy` tensor and the configured model name are coupled `Source: GitHub issue #84`.

### OpenAI-Compatible Server

`routellm/openai_server.py` launches a FastAPI server that mimics the `/v1/chat/completions` and `/v1/models` endpoints. It selects routers through the `--routers` CLI flag and reads model identifiers via `--strong-model` / `--weak-model` `Source: [routellm/openai_server.py:1-80]()`. Issue #63 shows a common deployment failure where `litellm.drop_params` is required to drop unsupported fields when forwarding to certain upstream providers (e.g., Ollama chat models) `Source: GitHub issue #63`.

### Package Surface and Dependencies

The top-level `routellm/__init__.py` re-exports `Controller` so callers can simply `from routellm import Controller` `Source: [routellm/__init__.py:1-20]()`. Runtime dependencies are declared in `pyproject.toml` and include `litellm` (for talking to many backends), `fastapi`, `uvicorn`, `transformers`, and `sentence-transformers` `Source: [pyproject.toml:1-60]()`.

## Request Flow and Integration

A typical request proceeds as follows: the client issues a chat completion request; the server or SDK passes the message list to the controller; the controller calls `router.acquire(...)`; the router returns `strong` or `weak`; the controller forwards the request to the corresponding model client via `litellm.completion(...)`; the response is returned unchanged to the caller `Source: [routellm/controller.py:80-140]()`. Because the surface mirrors OpenAI's, downstream tools — including LangChain via its `ChatOpenAI` adapter — integrate without modification, as confirmed by community discussion in issue #28 `Source: GitHub issue #28`.

## Known Operational Limitations

Community issues surface several recurring operational pain points: rate-limit and authorization errors when using hosted strong models (issue #78), compatibility errors with `litellm.drop_params` (issue #63), and the inability to use distinct base URLs for each side of the controller (issue #27). Issue #82 also notes that full training scripts are not bundled for every router variant. Together these define the current boundaries of the architecture and inform the roadmap for multi-model routing, embedded training code, and richer configuration.

---

<a id='page-routers'></a>

## Router Implementations & Algorithms

### Related Pages

Related topics: [Overview & System Architecture](#page-overview), [Deployment, Configuration & Threshold Calibration](#page-deployment), [Extensibility, Training & Custom Integrations](#page-extensibility)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [routellm/routers/routers.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/routers.py)
- [routellm/routers/matrix_factorization/model.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/matrix_factorization/model.py)
- [routellm/routers/matrix_factorization/train_matrix_factorization.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/matrix_factorization/train_matrix_factorization.py)
- [routellm/routers/similarity_weighted/utils.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/similarity_weighted/utils.py)
- [routellm/routers/similarity_weighted/generate_embeddings.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/similarity_weighted/generate_embeddings.py)
- [routellm/routers/causal_llm/model.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/causal_llm/model.py)
</details>

# Router Implementations & Algorithms

RouteLLM exposes a pluggable router abstraction whose sole responsibility is to decide, for a given user query, whether to forward the request to the **strong** model (e.g., GPT-4) or the **weak** model (e.g., Mixtral). All routers ultimately return a single binary win probability `p_win` against the strong model, which the controller then converts into a routing decision. Source: [routellm/routers/routers.py]()

## Router Registry and Common Interface

`routellm/routers/routers.py` acts as the central registry and dispatcher. It maps a string identifier (e.g., `"mf"`, `"sw"`, `"bert"`, `"causal_llm"`, `"random"`) to a concrete router class and exposes a uniform `get_router(router_name, **kwargs)` factory. Source: [routellm/routers/routers.py]()

Each router implements a consistent `calculate_cost` / `calculate_win_rate` style API so the OpenAI-compatible server (`routellm.openai_server`) can swap algorithms without changing upstream code. The registry is also the place where pre-trained artefacts (e.g., NumPy matrices, HuggingFace checkpoints) are loaded lazily on first use.

| Router ID | Algorithm Family | Trainable in Repo? |
|-----------|------------------|--------------------|
| `mf` | Matrix Factorization | Yes |
| `sw` / `sw_ranking` | Similarity-Weighted | Yes |
| `bert` | BERT classifier | **No** (training code missing) |
| `causal_llm` | Causal LM scoring | Partial (model only) |
| `random` | Baseline | N/A |

## Matrix Factorization Router

The MF router learns latent embeddings for queries and for the two models, then scores a query by the dot product with the strong-model vector. Source: [routellm/routers/matrix_factorization/model.py]()

The model is implemented as a thin PyTorch module holding two non-trainable buffers:

- `query_embeddings` – a pre-computed `[N_queries, D]` matrix loaded from a `.npy` artefact
- `model_embeddings` – a `[2, D]` matrix (one row per model) that *is* trained

Because query embeddings are fixed, training only updates `model_embeddings` via gradient descent on a Bradley–Terry-style loss. Source: [routellm/routers/matrix_factorization/train_matrix_factorization.py]()

Community context: the meaning of the `.npy` file and the choice to keep query vectors frozen are recurring questions (see issue #83). Issue #84 reports friction when swapping the embedding model away from the default; the trainer currently expects a specific embedding dimensionality produced upstream.

## Similarity-Weighted Router

The SW router avoids learned parameters and instead computes a *similarity-weighted* win-rate estimate over a pool of labelled calibration queries. Source: [routellm/routers/similarity_weighted/utils.py]()

The runtime path is:

1. Encode the incoming query with the configured embedding model (`generate_embeddings.py`).
2. Compute cosine similarity between the query and every calibration example.
3. Convert similarities into weights and aggregate the binary strong-model labels into a single win probability.

Source: [routellm/routers/similarity_weighted/generate_embeddings.py]()

This router is attractive for users who want a strong baseline without retraining, since the heavy artefact is just an embedding matrix plus labels.

## Causal LLM Router

The causal-LM router reformulates routing as a next-token prediction task. Given the user prompt, a small language model (e.g., a fine-tuned Llama variant) produces a token whose logit is interpreted as the strong-model win probability. Source: [routellm/routers/causal_llm/model.py]()

The module is responsible only for inference and token-to-probability decoding; the associated training pipeline lives outside the open-sourced repository, which mirrors the gap noted for the BERT router in issue #67.

## Known Gaps Reflected in Community Discussions

- **BERT training code missing** – issue #67 asks for the BERT training script; it is not present, only an inference hook in the registry. The same gap applies to the `causal_llm` trainer.
- **Custom embedding models for MF** – issue #84 reports errors when overriding the embedding model; the MF trainer hard-codes the dimensionality expected from the upstream pipeline.
- **Two-model assumption** – issue #23 highlights that every router above is fundamentally binary (strong vs. weak). The registry has no multi-arm interface today.

```mermaid
flowchart LR
    Q[User Query] --> R{router.calculate_win_rate}
    R -->|p_win >= threshold| S[Strong Model]
    R -->|p_win < threshold| W[Weak Model]
    S --> Resp[Response]
    W --> Resp
```

For most users the practical recommendation is: start with `mf` or `sw` (training scripts are present), and treat `bert` / `causal_llm` as inference-only checkpoints unless the upstream training artefacts are obtained separately.

---

<a id='page-deployment'></a>

## Deployment, Configuration & Threshold Calibration

### Related Pages

Related topics: [Overview & System Architecture](#page-overview), [Router Implementations & Algorithms](#page-routers)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [routellm/openai_server.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/openai_server.py)
- [routellm/calibrate_threshold.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/calibrate_threshold.py)
- [config.example.yaml](https://github.com/lm-sys/RouteLLM/blob/main/config.example.yaml)
- [examples/router_chat.py](https://github.com/lm-sys/RouteLLM/blob/main/examples/router_chat.py)
- [examples/python_sdk.md](https://github.com/lm-sys/RouteLLM/blob/main/examples/python_sdk.md)
- [examples/routing_to_local_models.md](https://github.com/lm-sys/RouteLLM/blob/main/examples/routing_to_local_models.md)
</details>

# Deployment, Configuration & Threshold Calibration

## Purpose and Scope

RouteLLM exposes three cooperating artefacts for bringing a router online: an OpenAI-compatible HTTP server for drop-in replacement of an OpenAI endpoint, a YAML configuration file that binds routers to model identifiers and endpoints, and a threshold-calibration tool that sets the operating point at which a query is sent to the strong model versus the weak model. Together they answer the operational questions *which model serves this request*, *which backend URL handles it*, and *how often the strong model is selected*. This page documents the deployment surfaces, the structure of `config.example.yaml`, and the calibration workflow that ties them together.

## OpenAI-Compatible Server Deployment

The primary deployment surface is `routellm/openai_server.py`, launched as a Python module:

```
python -m routellm.openai_server \
  --routers mf \
  --weak-model ollama_chat/codeqwen \
  --strong-model openai/gpt-4o-mini
```

Source: [routellm/openai_server.py:1-80](). The server speaks the standard `/v1/chat/completions` protocol, so any OpenAI client — including LangChain configured with an OpenAI-compatible `base_url` — can target it without code changes (relevant to issue #28 on LangChain integration).

Launch flags:

| Flag | Purpose |
|---|---|
| `--routers` | Comma-separated router names to register (`mf`, `bert`, `random`, `causal_llm`, `similarity_weighted`) |
| `--weak-model` / `--strong-model` | LiteLLM identifiers for the two backends |
| `--config` | Path to a YAML file overriding defaults (per-model `api_base`, threshold, embedding model) |

For deployments in which the strong and weak models live behind different OpenAI-compatible URLs — for example two local Ollama servers on different ports, or one local model and one hosted model — the `--config` flag loads a YAML file specifying distinct `api_base` values. Source: [config.example.yaml:1-40](). This directly addresses the community use case of routing between two Ollama instances (issue #76) and the request for separate base URLs per model (issue #27).

An alternative is the in-process Python SDK path documented in [examples/python_sdk.md](), where the `RouteLLM` controller is instantiated inside application code and no HTTP server is involved. This is preferable when the client and router share a process and HTTP overhead is undesirable.

## Configuration File Layout

`config.example.yaml` is the canonical template. Its top-level keys are:

- `router`: selects the router class (`mf`, `bert`, etc.) and its parameters — checkpoint path, embedding model identifier, hidden dimension.
- `strong_model` / `weak_model`: each entry holds a `model_name` (LiteLLM identifier) plus an optional `api_base` and `api_key` for non-default endpoints.
- `threshold`: a float (default `0.5`) used by routers that emit a continuous win-rate score; queries scoring at or above the threshold are routed to the strong model.

Source: [config.example.yaml:1-60](). The `router.embedding_model` key is the configuration knob for swapping the MF router's encoder, which is the change requested in issue #84. A minimal working configuration is shown in the example below.

```yaml
router:
  name: mf
  checkpoint_path: routellm/mf
  embedding_model: sentence-transformers/all-MiniLM-L6-v2

strong_model:
  model_name: openai/gpt-4o-mini
  api_base: https://api.openai.com/v1

weak_model:
  model_name: ollama_chat/llama3
  api_base: http://localhost:11434

threshold: 0.5
```

Source: [examples/router_chat.py:1-40](). The same structure is used by [examples/routing_to_local_models.md]() when both endpoints point at local servers.

## Threshold Calibration

Routers such as `mf` and `bert` emit a continuous score — the predicted probability that the strong model wins on a given query — which is then compared against the routing threshold. Because that threshold directly determines the cost/quality trade-off, it should be set empirically rather than left at its default.

`routellm/calibrate_threshold.py` sweeps candidate thresholds on a labelled evaluation set and reports the cost saving achieved at a fixed quality target (typically 95% of strong-model performance). Typical invocation:

```
python -m routellm.calibrate_threshold \
  --router mf \
  --config config.yaml \
  --target-quality 0.95
```

Source: [routellm/calibrate_threshold.py:1-60](). The script prints a recommended threshold, which is then written into `config.example.yaml` under `threshold:`. Re-calibration is recommended whenever the strong or weak model is swapped, since the predicted win-rate distribution shifts with model identity and a threshold tuned for one pair is suboptimal for another.

## Known Deployment Pitfalls

Community-reported issues surface several recurring deployment failures:

- `litellm.drop_params` errors on the OpenAI server (issue #63) usually trace to a LiteLLM version mismatch; pinning the version recorded in the repository's dependency list resolves it.
- Authorization errors with the BERT router (issue #78) occur when the Hugging Face checkpoint download is blocked; pre-downloading weights or setting `HF_TOKEN` is required for restricted environments.
- Rate-limit errors on hosted models (issue #78) indicate that the strong-model endpoint is being hit too often — lowering the calibrated threshold shifts traffic back to the weak model.

Source: [examples/routing_to_local_models.md:1-40]().

## Summary

Deployment in RouteLLM reduces to three artefacts: an OpenAI-compatible server for production traffic, a YAML configuration that binds routers to model endpoints and base URLs, and a calibration tool that sets the routing threshold. Keeping the strong/weak model identities, their `api_base` URLs, and the calibrated threshold in sync is the main operational responsibility, and re-calibration should accompany any change of model pair.

---

<a id='page-extensibility'></a>

## Extensibility, Training & Custom Integrations

### Related Pages

Related topics: [Router Implementations & Algorithms](#page-routers), [Deployment, Configuration & Threshold Calibration](#page-deployment)

<details>
<summary>Related Source Files</summary>

The following source files were used to generate this page:

- [routellm/routers/routers.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/routers.py)
- [routellm/routers/matrix_factorization/train_matrix_factorization.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/matrix_factorization/train_matrix_factorization.py)
- [routellm/controller.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/controller.py)
- [routellm/routers/similarity_weighted/generate_embeddings.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/routers/similarity_weighted/generate_embeddings.py)
- [routellm/openai_server.py](https://github.com/lm-sys/RouteLLM/blob/main/routellm/openai_server.py)
- [pyproject.toml](https://github.com/lm-sys/RouteLLM/blob/main/pyproject.toml)
- [examples/config.example.yaml](https://github.com/lm-sys/RouteLLM/blob/main/examples/config.example.yaml)
</details>

# Extensibility, Training & Custom Integrations

RouteLLM is designed as a modular routing framework where the core routing decision can be swapped, retrained, or replaced without rewriting the request pipeline. This page covers the extension points exposed by the router layer, the training workflows that ship in the repository, and the configuration hooks used to integrate custom models, embeddings, and external clients.

## Router Abstraction and Registry

All routing strategies inherit from a common base class defined in the routers package. The registry pattern lets a single server process load one or more routers at startup and dispatch incoming requests to the active strategy.

- `BaseRouter` defines the contract that every strategy must implement: an `acquire`/`acquire_batch` method that maps a conversation to a `(model, weight)` tuple, plus a `calculate_strong`/`calculate_weak` pair that returns win-rate style probabilities.
- The `ROUTERS` dictionary in `routellm/routers/routers.py` maps short CLI aliases (`mf`, `bert`, `random`, `causal_llm`, `similarity_weighted`) to concrete classes, enabling runtime selection. Source: [routellm/routers/routers.py:1-80]()
- Helper utilities for tokenization and message normalization are exposed through the same module so custom routers can reuse the prompt-construction logic instead of re-implementing it.

This registry is the primary extension point: to add a new router, you implement a subclass of `BaseRouter`, register it in `ROUTERS`, and reference the alias from the `--routers` CLI flag. Community issue #67 ("Can BERT router source code (esp. for training) be added to the repo?") and #82 ("Are training scripts available?") both reflect demand for visible training code for non-`mf` strategies, which is exactly what this registry pattern makes possible.

## Training Matrix Factorization and Similarity-Weighted Routers

The repository ships end-to-end training scripts for two learned routers. They share a data-preparation step but diverge at the model-fitting stage.

### Matrix Factorization (MF)

The script `routellm/routers/matrix_factorization/train_matrix_factorization.py` loads pre-computed query embeddings from a `.npy` file rather than encoding prompts at training time. This is intentional: the frozen embedding tensor (`query_embedding`) acts as a static lookup table of prompt features, while only the per-model weight vectors are updated by SGD. Community issue #83 ("What is the .npy file for query embedding in mf router training.") highlights that this file is the cached output of an upstream sentence-transformer run, which decouples the encoder from the router fit. Source: [routellm/routers/matrix_factorization/train_matrix_factorization.py:58-62]()

The training loop reads a battle dataset (model A vs. model B win/loss labels), looks up the corresponding row in the `.npy` embedding matrix, and minimizes a binary cross-entropy loss against the model strength indicators. The resulting weight matrix is serialized to disk and reloaded by `MatrixFactorizationRouter` at inference time.

### Similarity-Weighted

`routellm/routers/similarity_weighted/generate_embeddings.py` is a one-shot preprocessing utility: it encodes a calibration set of prompts with a sentence-transformer model and stores the vectors so the router can perform k-nearest-neighbor style strength estimation at request time. Because the embeddings are produced offline, the inference path stays constant-memory. Source: [routellm/routers/similarity_weighted/generate_embeddings.py:1-50]()

### Custom Encoders

Community issue #84 ("How can I use the different embedding model for mf router") asks whether alternatives like `intfloat/e5-small-v2` can be substituted. The answer hinges on regenerating the `.npy` file with the new encoder before invoking the training script — the router itself is encoder-agnostic as long as the dimensionality matches the model's weight matrix. This is the supported path for plugging in a domain-specific encoder.

## Controller Configuration and Custom Integrations

The `Controller` is the bridge between a routing decision and the downstream model client. It exposes hooks for plugging in different transport layers without touching the routers.

| Hook | Purpose | Default |
|------|---------|---------|
| `strong_model` / `weak_model` | LiteLLM model identifiers passed to the routing layer | GPT-4o / Llama 3.1 8B |
| `strong_model_url` / `weak_model_url` | Override base URL for one side | OpenAI default |
| `router` | Any registered router instance | `mf` |
| `openai_client` | Injected `openai.OpenAI` client used by the SDK path | Built lazily |

Source: [routellm/controller.py:1-120]()

Community issue #27 ("Allow Separate Base URLs for Strong and Weak Models in Controller") is addressed exactly by the `strong_model_url` / `weak_model_url` parameters — they allow a user to point the strong side at OpenAI while keeping the weak side on a self-hosted Ollama or vLLM endpoint.

### OpenAI-Compatible Server

`routellm/openai_server.py` exposes the same controller through an OpenAI-shaped HTTP endpoint, which is the integration surface most third-party tools (LangChain, LlamaIndex, raw `openai-python` clients) consume. A typical launch looks like:

```bash
python -m routellm.openai_server \
  --routers mf \
  --weak-model ollama_chat/llama3 \
  --config examples/config.example.yaml
```

The server reads `examples/config.example.yaml` to populate controller defaults, including model URLs and API keys. Community issue #76 ("Try to run 2 Ollama") demonstrates the same pattern with two local Ollama servers (Phi4 as strong, Llama3 8B as weak). Source: [routellm/openai_server.py:1-80]()

### LangChain and Other Clients

Issue #28 ("Langchain integration") asks how RouteLLM can be used with LangChain. Because the server is OpenAI-compatible, any client that accepts a `base_url` parameter can target it:

```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    base_url="http://localhost:6060/v1",
    api_key="EMPTY",
    model="route",
)
```

No LangChain-specific adapter is required; the OpenAI surface is the integration boundary.

## Limitations Reported by the Community

Several recurring friction points are documented in the issue tracker and are worth noting before extending RouteLLM:

- **Two-model ceiling.** Issue #23 requests support for more than two models. The current `Controller` and all shipped routers are built around a binary strong/weak decision; extending to N-way routing requires subclassing `BaseRouter` and adapting the controller's dispatch logic.
- **BERT training code.** Issue #67 notes that BERT router training code is not in the public repo. Practitioners who need it must train externally and load weights through a custom subclass.
- **Embedding mismatches.** Issue #84 shows that swapping the encoder fails silently unless the `.npy` is regenerated with matching dimensionality and the MF router's projection layer is re-initialized.
- **Server-side `litellm.drop_params` errors.** Issue #63 reports that certain model/provider pairs trigger `drop_params` errors in the server. This is a litellm compatibility issue, not a router issue, and is worked around by upgrading the litellm version pinned in `pyproject.toml`.

## Summary of Extension Workflows

| Goal | Where to start |
|------|----------------|
| Add a new routing algorithm | Subclass `BaseRouter`, register in `ROUTERS` |
| Train a new MF model | Replace `.npy` embeddings, run `train_matrix_factorization.py` |
| Use a custom encoder | Regenerate embeddings via `generate_embeddings.py` (or equivalent), then retrain |
| Point strong/weak at different providers | `strong_model_url` / `weak_model_url` on `Controller` |
| Use from LangChain / any OpenAI client | Launch `openai_server.py`, point client at the URL |

The framework's design keeps routing, training, and transport as independent layers, which is what makes the surface above stable even as new routers and providers are added.

---

<!-- evidence_pipeline_checked: true -->
<!-- evidence_injected: true -->

---

## Pitfall Log

Project: lm-sys/RouteLLM

Summary: Found 11 structured pitfall item(s), including 1 high/blocking item(s). Top priority: Configuration risk - Configuration risk requires verification.

## 1. Configuration risk - Configuration risk requires verification

- Severity: high
- Evidence strength: source_linked
- Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/63

## 2. Installation risk - Installation risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/87

## 3. Configuration risk - Configuration risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/76

## 4. Capability evidence risk - Capability evidence risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: README/documentation is current enough for a first validation pass.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: capability.assumptions | https://github.com/lm-sys/RouteLLM

## 5. Runtime risk - Runtime risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/84

## 6. Maintenance risk - Maintenance risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

## 7. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: downstream_validation.risk_items | https://github.com/lm-sys/RouteLLM

## 8. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: no_demo
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: risks.scoring_risks | https://github.com/lm-sys/RouteLLM

## 9. Security or permission risk - Security or permission risk requires verification

- Severity: medium
- Evidence strength: source_linked
- Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: community_evidence:github | https://github.com/lm-sys/RouteLLM/issues/78

## 10. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: issue_or_pr_quality=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

## 11. Maintenance risk - Maintenance risk requires verification

- Severity: low
- Evidence strength: source_linked
- Finding: release_recency=unknown。
- User impact: May increase setup, validation, or first-run risk for the user.
- Evidence: evidence.maintainer_signals | https://github.com/lm-sys/RouteLLM

<!-- canonical_name: lm-sys/RouteLLM; human_manual_source: deepwiki_human_wiki -->