Related Pages

Related topics: System Architecture and Core Components, Model Integrations and Inference Backends

Introduction and Getting Started with Outlines

1. Purpose and Philosophy

Outlines is a Python library for structured generation with large language models (LLMs). Rather than parsing, regex-matching, or post-processing free-form text, Outlines constrains token sampling *during* generation so the produced output is guaranteed to match a target schema. According to the README.md, the library's design goals are:

• Provider independence — the same code runs against OpenAI, Ollama, vLLM, transformers, llama.cpp, and more.
• Native Python ergonomics — output structure is described using Literal, int, Pydantic models, regex patterns, or context-free grammars, mirroring Python's own type system.
• Guaranteed validity — invalid tokens are masked at each decoding step, eliminating broken JSON and fragile parsers.

The maintainer .txt frames structured generation as a reliability primitive: "Outlines guarantees structured outputs during generation — directly from any LLM." (Source: README.md)

2. Model Integrations and Routing

Outlines groups its providers in outlines/models/ and exposes routing helpers so user code never imports a concrete client class directly. The __init__.py enumerates the supported backends (outlines/models/__init__.py:1-44):

Each module follows the same pattern: a <Provider>TypeAdapter formats prompts and output types, and a from_<provider> factory wraps the underlying client. For example, the Dottxt adapter validates that the input is a str and serializes output_type to a JSON schema string (outlines/models/dottxt.py:24-60). Similarly, the vLLM offline engine builds a SamplingParams object from the requested output type before calling LLM.generate or LLM.chat (outlines/models/vllm_offline.py:1-60).

flowchart LR
    A[User prompt + output_type] --> B[TypeAdapter]
    B --> C{Provider}
    C -->|white-box| D[LogitsProcessor / FSM]
    C -->|HTTP API| E[response_format kwarg]
    C -->|Offline engine| F[SamplingParams]
    D --> G[Token-by-token generation]
    E --> G
    F --> G
    G --> H[Structured output]

3. Quickstart Pattern

The README introduces a uniform invocation pattern: model(prompt, output_type). A representative transformers example (README.md):

import outlines
from transformers import AutoTokenizer, AutoModelForCausalLM
from typing import Literal

model = outlines.from_transformers(
    AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),
    AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),
)

result = model("Is this review positive? 'It was a great meal'", Literal["Positive", "Negative", "Neutral"])
print(result)  # "Positive"

The same call shape works on remote providers. The AsyncMistral model streams tokens while delegating response_format to the Mistral client (outlines/models/mistral.py). The SGLang wrapper adds structured-output arguments to client.chat.completions.create and yields delta content chunks (outlines/models/sglang.py). TGI uses singledispatchmethod to handle multiple input shapes and converts DSL types (regex, JSON schema, CFG) to a structured-output payload (outlines/models/tgi.py).

4. Output Types and Known Limitations

The library supports a layered set of output type families, each documented in the README's *Core Features* and *Other Features* tables (README.md):

• Multiple choices — Literal["Yes", "No"]
• Basic Python types — int, float, bool, date
• JSON / Pydantic — full nested models, Union types, and Enum values
• Regex patterns — arbitrary regular expressions
• Context-free grammars — Lark/EBNF grammars for complex languages
• Prompt templates — Jinja-based reusable prompts via outlines.Template
• Applications — wrappers that bind a template and output type to a single callable

A recurring community pain point is incomplete JSON schema coverage. Issue #215 tracks the gradual implementation of minLength, pattern, and standard string formats — only maxLength was initially supported. The latest release notes (v1.3.0) also highlight ongoing work on uniform exception classes across model providers (#1823) and clearer error messages when model type detection fails (#1862). Long-running feature requests include probability distribution access for outlines.generate.choice (#479), LangChain integration (#344), and mlx-vlm support for vision models (#1188).

A practical failure mode worth noting: tokenizers with multi-byte or replacement characters (e.g., Qwen1.5, Phi-2) can raise RuntimeError: Cannot convert token in the regex FSM (#820). When a model is not yet in the integration matrix, the workaround is to fall back to a black-box provider that accepts a response_format argument.

See Also

• Outlines core types and DSL: outlines/types/dsl.py
• Regex guided generation internals: outlines/fsm/regex.py
• vLLM offline batched generation: outlines/models/vllm_offline.py
• Mistral streaming integration: outlines/models/mistral.py
• Dottxt hosted API: outlines/models/dottxt.py

Related Pages

Related topics: Introduction and Getting Started with Outlines, Model Integrations and Inference Backends, Output Types, Templates, and Community Requests

System Architecture and Core Components

Overview

Outlines is a Python library that guarantees structured outputs from large language models during generation itself, rather than trying to repair bad outputs after the fact. The library follows a single guiding pattern: pass a desired Python output type to a model and the result will conform to that type exactly (Literal["Yes", "No"], int, a Pydantic model, etc.) Source: [README.md].

Architecturally, Outlines is built as a thin, provider-agnostic abstraction layer on top of a wide range of LLM backends. The same high-level API runs across OpenAI, Gemini, vLLM, Ollama, llama.cpp, MLX, Transformers, SGLang, and the .txt API, so users can switch models without rewriting application code Source: [README.md]. The library is published as outlines on PyPI and is developed by .txt.

Core Architecture: Model Abstraction Layer

The internal architecture is organised by provider rather than by capability (completion, chat, diffusers, etc.). The top-level outlines/models/__init__.py module is a single routing surface that re-exports one wrapper class and one from_* factory function per provider Source: [outlines/models/__init__.py].

flowchart TD
    A[User Code: model(prompt, output_type)] --> B[Outlines Model Wrapper]
    B --> C[ModelTypeAdapter]
    C --> D1[format_input]
    C --> D2[format_output_type]
    B --> E{Provider Class}
    E --> F[Steerable Models<br/>Transformers / LlamaCpp / MLXLM]
    E --> G[Black-Box Models<br/>OpenAI / Anthropic / Gemini / Ollama / vLLM / SGLang / TGI / LMStudio / Dottxt / Mistral]
    F --> H[Logits Processors / FSM]
    G --> I[Server-side structured output]

Two high-level type aliases summarise the implementation strategy:

Source: outlines/models/__init__.py

The Model and ModelTypeAdapter base classes — imported from outlines.models.base — define the contract that every provider must satisfy: a format_input method to coerce prompts and chat payloads, and a format_output_type method to translate an Outlines type (CFG, JsonSchema, Regex, …) into something the backend understands Source: [outlines/models/__init__.py].

Provider Integrations and the Adapter Pattern

Every backend ships its own *TypeAdapter subclass of ModelTypeAdapter. For example, the .txt API adapter rejects non-str inputs explicitly and converts output types to JSON schema strings before sending them to the server Source: [outlines/models/dottxt.py]:

def format_output_type(self, output_type: Optional[Any] = None) -> str:
    if output_type is None:
        raise TypeError("You must provide an output type ...")
    # serialised as a JSON schema string for the upstream API

SGLang follows the same pattern but is designed for streaming through an OpenAI-compatible chat-completions endpoint. Its _build_client_args method merges the formatted messages, the output type configuration, and any user-supplied inference_kwargs, then hands them to client.chat.completions.create Source: [outlines/models/sglang.py].

The Mistral integration additionally supports streaming and async streaming. AsyncMistral reformats the user input into chat messages, asks the adapter for the structured response_format, and then iterates client.chat.stream_async while yielding delta.content chunks as they arrive Source: [outlines/models/mistral.py].

For the Gemini backend, the factory from_gemini simply wraps a google.genai.Client in the Outlines Gemini model class, exposing .text-style access to streamed chunks Source: [outlines/models/gemini.py].

Local inference is handled by Transformers and VLLMOffline. The Transformers wrapper calls a logits_processor derived from the requested output type and forwards the result to model.generate, taking care to handle multi-modal models whose generate returns a 2-D tensor even when num_return_sequences == 1 Source: [outlines/models/transformers.py]. The vLLM offline path branches on input type: chat payloads go through model.chat while raw prompts go through model.generate, both using sampling parameters built from the supplied logits processor Source: [outlines/models/vllm_offline.py].

Error Handling, Extensibility, and the v1.3.0 Release

As of Outlines v1.3.0, all provider integrations route their native exceptions through a single normalize_provider_errors context manager, giving users uniform exception classes across model providers Source: [README.md — Release notes, PR #1823]. This is why every provider module opens its network call with with normalize_provider_errors(PROVIDER): — for example, the Dottxt, Mistral, and SGLang wrappers all follow this idiom Source: [outlines/models/dottxt.py, outlines/models/mistral.py, outlines/models/sglang.py].

The same release also fixed error messages for model type determination, contributed by a first-time community contributor Source: [README.md — PR #1862].

Several open community discussions map directly onto the architecture:

• JSON schema field constraints (issue #215): the JsonSchema → adapter path currently supports only a subset of the JSON Schema spec; expanding it requires changes in the type DSL and the per-provider adapters.
• Tokenizer issues with llama.cpp (issue #820): the SteerableModel path encodes regexes into finite state machines (outlines/fsm/regex.py); some model vocabularies produce tokens that cannot be converted to bytes and crash the encoder.
• Probability distributions for generate.choice (issue #479): the SteerableModel path exposes token-level logits and is the natural place to surface log-probabilities alongside constrained outputs.
• MLX-VLM support (issue #1188): extending the local SteerableModel family beyond MLXLM to vision-language models.

Together these illustrate that the abstraction layer is intentionally narrow: routing in outlines/models/__init__.py, an adapter in each provider file, and shared error normalization. Adding a new backend means writing one file with one from_* factory, one Model subclass, and one ModelTypeAdapter subclass — and the rest of the library, including prompts, templates, and output types, can be reused unchanged.

See Also

• Model Integrations — official documentation of supported backends.
• Output Types — covers the Literal, JSON schema, regex, and CFG types that flow through the adapter layer.
• Community / Contribute — guidance for adding a new provider integration.

Related Pages

Related topics: Introduction and Getting Started with Outlines, System Architecture and Core Components, Output Types, Templates, and Community Requests

Model Integrations and Inference Backends

Overview and Purpose

Outlines provides a uniform Python interface for structured (guided) generation across a broad range of LLM providers. The outlines.models package groups integrations by provider rather than by capability (completion, chat, batch, etc.), and exposes a single Model/AsyncModel abstraction so user code stays identical when switching backends.

The package re-exports both concrete model classes and from_<provider>() factory functions from per-provider submodules. These are organized into two practical categories by the top-level __init__.py:

• Steerable models — local runtimes that support token-level guided generation through Outlines' FSM/logits-processor machinery: LlamaCpp, MLXLM, and Transformers (including TransformersMultiModal). Source: outlines/models/__init__.py:43
• Black-box models — providers where structured output is delegated to the server's own guided-decoding engine: Anthropic, Dottxt, Gemini, LMStudio, Ollama, OpenAI, SGLang, TGI, VLLM, VLLMOffline, and Mistral. Source: outlines/models/__init__.py:44

A second generation of type adapters (OpenAITypeAdapter, MistralTypeAdapter, VLLMTypeAdapter, etc.) normalizes the prompt/chat input and the output_type constraint into the JSON Schema, regex, or CFG payload each remote API expects.

Architecture and Core Abstractions

flowchart TB
    User[User Code: model(prompt, output_type)] --> Model[Model / AsyncModel]
    Model --> Adapter[ModelTypeAdapter]
    Adapter --> Input[format_input -> messages / prompt]
    Adapter --> Output[format_output_type -> JSON Schema / regex]
    Input --> Backend[(Inference Backend)]
    Output --> Backend
    Backend --> Model
    Model --> Result[Validated structured result]

Every integration inherits from Model (or AsyncModel) defined in outlines/models/base.py. The base class wires together three responsibilities:

1. Holding the underlying client (e.g. an openai.OpenAI, vllm.LLM, or transformers pipeline object).
2. Delegating input/output-type translation to a ModelTypeAdapter.
3. Wrapping every provider call with normalize_provider_errors(...) from outlines.exceptions so failures surface as Outlines' uniform exception hierarchy. This is the work referenced in v1.3.0's "Implement uniform exceptions classes across model providers" change.

Source: outlines/models/base.py, outlines/models/openai.py, outlines/models/vllm.py

Local Steerable Backends

Transformers

from_transformers(model, tokenizer) wraps a Hugging Face AutoModelForCausalLM and AutoTokenizer. Outlines hooks the guided FSM into the model's generate() call via a logits processor, which is why this backend is in the SteerableModel union. A TransformersMultiModal variant extends the same machinery to vision-language checkpoints. Source: outlines/models/__init__.py:32-40

LlamaCpp

LlamaCpp binds to a llama_cpp.Llama instance and applies guided decoding at the sampler level. Community issue #820 ("Cannot convert token … (29333) to bytes") traces a tokenizer-compatibility defect to this path — some GGUF vocabularies (e.g. Qwen1.5, Phi-2) expose partial-byte tokens that the regex FSM cannot decode, while vocabularies such as OpenHermes-2.5-Mistral-7B work cleanly. Source: outlines/models/__init__.py:9, community issue #820.

MLXLM

MLXLM is the steerable backend for Apple Silicon, built on the mlx-lm library. Community issue #1188 requests a parallel mlx-vlm integration for vision models; today, MLX vision support is not in the public SteerableModel union, and users on macOS wanting multimodal still route through TransformersMultiModal. Source: outlines/models/__init__.py:43, community issue #1188.

Remote / Black-Box Backends

For remote providers, Outlines cannot touch the sampling loop, so it translates the requested output_type into the provider's native structured-output argument and lets the server enforce the schema.

Sources: outlines/models/openai.py, outlines/models/vllm.py, outlines/models/vllm_offline.py, outlines/models/mistral.py, outlines/models/sglang.py, outlines/models/tgi.py, outlines/models/dottxt.py.

Choosing a Backend and Common Pitfalls

Selection rule of thumb. If you need to keep the model in-process and want the most complete support for regex, JSON Schema, and CFG constraints, prefer Transformers (or VLLMOffline). If you want server-side scaling, use VLLM or SGLang. For commercial APIs, use the matching from_<provider>() factory and pass your existing client object.

Chat vs. raw prompts. OpenAITypeAdapter, VLLMTypeAdapter, and SGLang accept Chat objects, lists of messages, or strings; DottxtTypeAdapter and TGITypeAdapter are string-only and will raise TypeError on anything else. Source: outlines/models/dottxt.py:32-49, outlines/models/tgi.py:38-50.

Output-type availability. Not every structured type is supported by every remote provider. The Dottxt adapter refuses output_type=None, whereas OpenAI and vLLM allow unconstrained generation. Cross-provider tools (issue #344 — "Add Outlines in LangChain") depend on this adapter layer behaving consistently.

Probability distributions. Community issue #479 asks for log-probability return values from generate.choice. Today the return type from the Model class is text (or a list of texts) only; probability metadata is not exposed by the standard generate path.

Error handling. As of v1.3.0, every provider call is wrapped with normalize_provider_errors(PROVIDER), so a malformed schema or a network failure in any backend raises a single, consistent Outlines exception class with the provider name attached.

See Also

• Core Features (JSON Schema, regex, CFG, multiple choice)
• Output Types and Type Adapters
• Structured Generation Internals (FSM and logits processors)

Related Pages

Related topics: Introduction and Getting Started with Outlines, System Architecture and Core Components, Model Integrations and Inference Backends

Output Types, Templates, and Community Requests

Overview

Outlines is a Python library that guarantees structured outputs from large language models directly during generation rather than parsing outputs post-hoc. The library follows a philosophy that mirrors Python's own type system: a user specifies the desired output type and Outlines ensures the generated data matches that structure exactly. Source: README.md

The library supports four primary output type categories: Multiple Choices (Literal), JSON/Pydantic schemas, Regular Expressions, and Context-Free Grammars. It also provides reusable Jinja-based prompt templates and an application abstraction that encapsulates templates together with types into callable functions. Source: README.md

Output Types Architecture

The output type system is implemented as a ModelTypeAdapter abstraction shared across all model integrations. Each provider implements two methods, format_input and format_output_type, which transform Outlines-native types into the format expected by the underlying client. Source: outlines/models/base.py

flowchart LR
    A[User Output Type] --> B[ModelTypeAdapter.format_output_type]
    B --> C{Steerable or Black-Box?}
    C -->|Steerable| D[Logits Processor]
    C -->|Black-Box| E[response_format / format kwarg]
    D --> F[Transformers / vLLM / llama.cpp]
    E --> G[OpenAI / Mistral / Ollama / Dottxt]

Steerable models (where token logits can be inspected) include LlamaCpp, MLXLM, and Transformers. Black-box API providers include Anthropic, Dottxt, Gemini, LMStudio, Ollama, OpenAI, Mistral, SGLang, TGI, and VLLM. Source: outlines/models/__init__.py

For steerable models, the output type becomes a logits processor that constrains generation token-by-token. For example, the Transformers integration accepts an OutlinesLogitsProcessor via the format_output_type method and passes it into the model's generate call. Source: outlines/models/transformers.py

For black-box providers, the adapter formats the type as a JSON schema string passed to the underlying client. The Dottxt adapter, for example, raises a TypeError if no output type is supplied, and otherwise returns a serialized JSON schema. Source: outlines/models/dottxt.py

The Mistral adapter uses response_format as the chat completion parameter, while the SGLang adapter merges formatted output type arguments directly into the inference kwargs. Source: outlines/models/mistral.py, outlines/models/sglang.py

The Ollama integration supports JSON-schema-compatible output types only; it does not support regex or CFG constraints. It also does not support batch inference, raising NotImplementedError when generate_batch is invoked. Source: outlines/models/ollama.py

The vLLM offline integration uses sampling_params populated by the output type and dispatches to either model.chat or model.generate depending on whether the input is a list of messages or a string prompt. Source: outlines/models/vllm_offline.py

Prompt Templates

Outlines includes a Template class that supports both string-defined and file-loaded Jinja templates. Templates separate complex prompts from business logic and enable dynamic prompt generation including few-shot learning strategies. Templates can be constructed via outlines.Template.from_string or outlines.Template.from_file. Source: README.md

Community Requests and Known Issues

The community has raised several recurring feature requests and bug reports that influence the development roadmap.

Issue #215 tracks the implementation of remaining JSON schema constraints. As of v1.3.0, only maxLength for strings is fully implemented; minLength and pattern remain on the roadmap. Source: README.md

Issue #820 reports a RuntimeError: Cannot convert token failure inside the regex finite-state machine used by steerable models. The error originates in outlines/fsm/regex.py and affects model vocabularies where certain token IDs cannot be mapped to valid UTF-8 bytes. Source: outlines/models/transformers.py

Issue #479 requests exposure of token-level probabilities alongside constrained outputs, useful for evaluating model certainty. The current outlines.generate.choice API returns only the selected token, and surfacing log-probabilities requires extending the Model.generate signature. Source: outlines/models/base.py

The most recent release, Outlines v1.3.0, includes a fix for error messages during model type determination and introduces uniform exception classes across model providers via normalize_provider_errors. Source: README.md, outlines/models/dottxt.py

See Also

• Model Integrations
• Structured Generation Concepts
• Templates and Applications

Doramagic Pitfall Log

Found 10 structured pitfall item(s), including 2 high/blocking item(s). Top priority: Maintenance risk - Maintenance risk requires verification.

1. Maintenance risk: Maintenance risk requires verification

• Severity: high
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/dottxt-ai/outlines/issues/1303

2. Security or permission risk: Security or permission risk requires verification

• Severity: high
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/dottxt-ai/outlines/issues/1874

3. Configuration risk: Configuration risk requires verification

• Severity: medium
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/dottxt-ai/outlines/issues/1859

4. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/dottxt-ai/outlines

5. Runtime risk: Runtime risk requires verification

• Severity: medium
• Finding: Project evidence flags a runtime risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/dottxt-ai/outlines/issues/1868

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/dottxt-ai/outlines

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/dottxt-ai/outlines

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/dottxt-ai/outlines

9. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/dottxt-ai/outlines

10. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/dottxt-ai/outlines