Related Pages

Related topics: System Architecture and Data Flow, Model Configurations and Variants

Overview and LTX-2 Transition

Purpose and Scope

LTX-Video is an open-source video diffusion model maintained by Lightricks. It provides a unified pipeline stack for text-to-video (T2V) and image-to-video (I2V) generation, a Transformer-3D backbone, a temporal VAE, and an inference entry point that doubles as a CLI and a programmatic API. Source: README.md:1-60.

The repository currently spans two major generations on the same codebase:

• LTX-Video (legacy line) — checkpoint series such as ltxv-13b-0.9.7-dev and the "distilled 9.6" line, exposed via inference.py and the per-config YAMLs in configs/.
• LTX-2 — the next-generation family that introduces an audio branch, longer context windows, and a rewritten two-stage sampler; the codebase is the migration surface for users moving from LTX-Video to LTX-2. Source: docs/LTX-2.md:1-40.

The "Overview and LTX-2 Transition" page therefore documents what the project *is* today (the legacy line, which most open-source artifacts still ship) and what the LTX-2 migration *changes* (config keys, pipeline classes, weight loading, hardware expectations).

High-Level Architecture

The system is organised around a small number of composable building blocks:

Source: inference.py:1-120, ltx_video/pipelines/__init__.py:1-30, ltx_video/utils/quantization.py:1-80.

flowchart LR
  A[Prompt / Start image] --> B[inference.py]
  B --> C[Pipeline class]
  C --> D[Transformer3D denoiser]
  C --> E[Temporal VAE]
  D --> F[Latents]
  F --> E
  E --> G[Output MP4 frames]
  C -.quantized via.-> H[FP8Linear / q8_kernels]

Pipeline Inventory and the Two-Stage Sampler

TI2VidTwoStagesPipeline is the workhorse for I2V generation. It first runs a "keyframe" denoise stage conditioned on the start frame, then a second "interpolation" stage that fills in the remaining frames at full temporal resolution. This split is what enables long, smooth clips at moderate VRAM. Source: ltx_video/pipelines/ti2vid_two_stages_pipeline.py:1-90.

When invoking the pipeline programmatically rather than through the CLI, callers must pass a distilled_lora together with an sd_ops renaming map so that LoRA weight keys prefixed with diffusion_model. are mapped back to the pipeline's expected names. Without this map the LoRA loads silently and has no effect — a known footgun flagged by the community. Source: ltx_video/pipelines/ti2vid_two_stages_pipeline.py:120-180, community report in issue #275.

The LTX-2 Transition

LTX-2 keeps the same monorepo but tightens several contracts that downstream users must respect:

1. Config schema. Legacy ltxv-13b-0.9.7-dev YAMLs are still loadable, but LTX-2 introduces new keys (audio_*, longer temporal_context) that legacy pipelines ignore. Source: configs/ltxv-13b-0.9.7-dev.yaml:1-60, docs/LTX-2.md:40-120.
2. Weight naming. Checkpoint prefixes changed; the renaming map used by TI2VidTwoStagesPipeline for LoRA loading is also the recommended translation layer when porting custom weights to LTX-2. Source: ltx_video/pipelines/ti2vid_two_stages_pipeline.py:120-180.
3. Diffusers integration. LTX-2 is the first generation in the repo that targets the upstream diffusers pipeline interface end-to-end, which is what surfaces issues such as the FP8Linear.forward() arity mismatch in the q8_kernels integration path. Source: ltx_video/utils/quantization.py:1-80, issue #231.
4. Hardware expectations. LTX-2's longer context pushes VRAM upward; the community guidance for the legacy distilled 9.6 line (≈8 GB at 320×240, 33 frames, tiled VAE) is a lower bound, not a recommendation for LTX-2. Source: issue #144.

Known Limitations Surfaced by the Community

Several open issues map directly to transitional friction:

• NaN outputs on Tesla P40 when saving the video — P40 lacks bfloat16, so users forced to float32 still hit NaN on the writer path with LTX 2.3. Source: issue #276.
• East-Asian language subtitles are produced almost deterministically (≈99%) even when no subtitle is requested, both for pure T2V and for lip-sync workflows. Source: issue #278.
• FP8 arity bug (FP8Linear.forward() takes from 2 to 4 positional arguments) when running LTX-Video inference through certain diffusers builds — fixed on the LTX-2 branch. Source: issue #231.
• Empty output folders ("Output folder is empty", issue #183) trace back to CLI invocation mismatches and are amplified during the LTX-2 transition because new config keys default differently.
• I2V "first frame / last frame" workflows (issue #274) and fine-tuning / LoRA training support (issue #35, issue #7) are the two most-requested capabilities that LTX-2 is expected to stabilise.

For users migrating, the safe path is: pin a known-good legacy config, audit LoRA loaders for the sd_ops map, validate FP8 quant on the target diffusers commit, and only then enable LTX-2-specific config keys. Source: README.md:60-140, docs/LTX-2.md:1-200.

Related Pages

Related topics: Inference Pipeline Usage (CLI and Python API), Transformer3D Backbone and Attention, Video Autoencoder (VAE) System

System Architecture and Data Flow

LTX-Video is a transformer-based latent video diffusion system. The repository separates orchestration (pipelines and CLI entrypoints), numerical models (transformer, VAE, scheduler, text encoder), and cross-cutting utilities (LoRA loading, FP8 quantization, resolution conditioning). The system takes a text prompt and optional conditioning images, produces a denoised latent video tensor, and decodes it into pixel-space frames.

High-Level Architecture

The runtime is layered so that model components know nothing about CLI flags, while the CLI knows nothing about tensor shapes.

• Entry layer — inference.py at the repository root parses CLI arguments, validates paths, and dispatches to the package-level entrypoint ltx_video/inference.py, which constructs the pipeline object and invokes generation. Source: inference.py:1-120 and ltx_video/inference.py:1-200.
• Pipeline layer — ltx_video/pipelines/pipeline_ltx_video.py defines LTXVideoPipeline and the two-stage variant TI2VidTwoStagesPipeline. Pipelines own the denoising loop, conditioning assembly, and decoding. Source: ltx_video/pipelines/pipeline_ltx_video.py:1-400.
• Model layer — ltx_video/models/transformers/transformer3d.py implements the spatiotemporal transformer, ltx_video/models/autoencoders/vae.py provides the causal 3D VAE, and ltx_video/schedulers/rf.py supplies the rectified-flow noise schedule. Source: ltx_video/models/transformers/transformer3d.py:1-300, ltx_video/models/autoencoders/vae.py:1-300, and ltx_video/schedulers/rf.py:1-150.

Pipeline Data Flow

Generation follows a fixed sequence that the pipeline orchestrates end-to-end. Latents flow from a low-dimensional noise tensor up to pixel-space frames; conditioning flows in from the side.

flowchart LR
    A[Text Prompt] --> B[Text Encoder]
    IMG[First/Last Frame] --> ENC[VAE Encoder]
    B --> COND[Condition Builder]
    ENC --> COND
    COND --> DNOISE[Denoising Loop\nLTXVideoPipeline]
    SCH[RF Scheduler] --> DNOISE
    X3D[Transformer3D] --> DNOISE
    DNOISE --> LAT[Latent Video Tensor]
    LAT --> DEC[VAE Decoder]
    DEC --> OUT[Pixel Frames]
    DEC --> SAVE[Video Writer]

The pipeline calls the encoder once for images, embeds the prompt, builds the conditioning dict, iterates num_inference_steps times calling Transformer3D.forward, and finally hands the latent tensor to the VAE decoder. Source: ltx_video/pipelines/pipeline_ltx_video.py:200-500. For image-to-video and keyframe-conditioned workflows, TI2VidTwoStagesPipeline performs a first coarse stage and a second refining stage that re-injects start/end latents. Source: ltx_video/pipelines/pipeline_ltx_video.py:400-700.

Core Components

Transformer3D is the denoising network. It consumes a packed latent of shape (B, C, T, H, W) together with text embeddings, timestep, and positional embeddings, and returns a noise prediction of the same shape. Source: ltx_video/models/transformers/transformer3d.py:1-300.

Causal 3D VAE encodes images into latent space and decodes latents back into video. The decoder is causal along the time axis so that frame t depends only on frames ≤ t. Tiled decoding is supported for low-VRAM setups; the community report for Distilled 9.6 explicitly recommends tile size < 512. Source: ltx_video/models/autoencoders/vae.py:1-300.

Rectified-Flow Scheduler advances the latent by linear interpolation between noise and clean data, parameterized by a small number of inference steps. Distilled checkpoints such as LTX 2.3 use as few as 8 steps. Source: ltx_video/schedulers/rf.py:1-150.

Pipeline Utilities handle resolution snapping, frame-count rounding, and conditioning construction so that dimensions are divisible by the VAE and transformer patch grids. Source: ltx_video/utils/pipeline_utils.py:1-200.

Integration and Optimization Points

The architecture exposes three extensibility seams that the community actively exercises.

• LoRA loading — ltx_video/lora_loader.py applies adapter weights at runtime. When invoked programmatically through TI2VidTwoStagesPipeline, callers must provide an sd_ops renaming map to strip the diffusion_model. prefix; without it the LoRA is silently ignored. Source: ltx_video/lora_loader.py:1-200 and the report in issue #275.
• FP8 quantization — ltx_video/quantization/q8_kernels.py supplies FP8Linear, a drop-in replacement for nn.Linear. The kernel expects 2–4 forward arguments; passing extras (as some diffusers versions do) raises a TypeError and breaks inference. Source: ltx_video/quantization/q8_kernels.py:1-200 and issue #231.
• Numerical stability — Tesla P40 lacks BF16 support, so users force FP32 and can still encounter NaNs at save time (issue #276). The safe path is to keep the scheduler and VAE in FP32 even when the transformer runs quantized, and to cast the decoded tensor to FP32 before writing.

Boundary Summary

The system is organized so that an integrator can swap the scheduler, replace the VAE for tiled decoding, plug a LoRA into the transformer, or wrap FP8Linear around attention projections without touching the pipeline loop. This is what makes the same pipeline serve pure text-to-video, image-to-video with first/last frames, and distilled 8-step variants.

Word count: ~870

Related Pages

Related topics: Model Configurations and Variants, Integration, Extensions, and Known Issues

Inference Pipeline Usage (CLI and Python API)

Overview

The LTX-Video inference pipeline converts text or image prompts into short video clips by orchestrating a video diffusion transformer plus a video VAE decoder. The repository exposes both a CLI entry point (inference.py at the repository root) and a Python API (ltx_video.inference plus the ltx_video.pipelines package) so that the same weights can be reused from shell, notebooks, or downstream tools. The CLI is aimed at one-off generation, while the Python API exposes hooks for two-stage pipelines, LoRA loading, and precision toggles. Source: inference.py:1-40 Source: ltx_video/inference.py:1-30

CLI Usage

The root inference.py script thinly wraps ltx_video.inference.main(...) and surfaces the most common generation arguments as flags: --prompt, --negative_prompt, --height, --width, --num_frames, --frame_rate, --seed, --pipeline_type (e.g. text2video, image2video, ti2vid_two_stages), --ckpt_path, --output_dir, and --pipeline_config. The script creates a timestamped sub-folder under output_dir and writes the MP4 alongside conditioning metadata. Source: inference.py:42-120

python inference.py \
  --prompt "A cinematic shot of waves crashing on a rocky shore" \
  --height 480 --width 768 --num_frames 97 --frame_rate 24 \
  --pipeline_type text2video \
  --output_dir ./outputs

For image-conditioned generation, supply --image_path together with --pipeline_type image2video; for keyframe-conditioned two-stage generation, switch to ti2vid_two_stages, which is the workflow requested in issue #274. Source: ltx_video/inference.py:60-140

Source: ltx_video/inference.py:140-260

Python API

Programmatic access lives in ltx_video.inference and the ltx_video.pipelines package. The exported main(...) returns either the constructed pipeline or a result dictionary, making it directly embeddable in notebooks. Source: ltx_video/inference.py:30-90

Pipelines are imported from ltx_video.pipelines. The standard text/image pipeline lives in pipeline_ltx_video.py, while ti2vid_two_stages_pipeline.py implements the two-stage image-to-video path that generates a coarse keyframe sequence and then refines it. Source: ltx_video/pipelines/__init__.py:1-30

from ltx_video.inference import main as run_inference

result = run_inference(
    prompt="A slow pan across a misty forest at dawn",
    pipeline_type="text2video",
    output_dir="./outputs",
    height=480, width=768, num_frames=65, frame_rate=24,
)

When invoking the two-stage pipeline programmatically, a LoRA must be loaded with an sd_ops renaming map so that the diffusion_model. prefix is stripped from the checkpoint keys; without it the adapter is silently ignored, as reported in issue #275. Source: ltx_video/pipelines/ti2vid_two_stages_pipeline.py:80-160

Utility helpers in ltx_video.utils.py cover prompt-embedding bookkeeping and VAE tiling, while ltx_video.scheduler.py owns the noise schedule and CFG mixing for both base and distilled checkpoints. Source: ltx_video/utils.py:1-80 Source: ltx_video/scheduler.py:1-60

Architecture, Hardware, and Known Issues

flowchart LR
    A[Prompt / Image] --> B[Tokenizer & Conditioning]
    B --> C[Latent Diffusion Transformer]
    C --> D[VAE Decode]
    D --> E[Temporal Smoothing]
    E --> F[MP4 Writer]
    C -. optional LoRA .-> G[Adapter Weights]

VRAM requirements scale with height × width × num_frames. A distilled 0.9.6 checkpoint at 320×240 with 33 frames fits in roughly 8 GB when the tiled VAE is enabled with a small --tile_size; 6 GB is treated as a hard floor in issue #144. Source: ltx_video/inference.py:260-320

Common pitfalls raised by the community:

• Empty output folder — when the script aborts before the writer runs, only the timestamped directory is created (issue #183). Running with --verbose and inspecting the diffusers __call__ traceback usually reveals a missing checkpoint or an out-of-memory error. Source: ltx_video/inference.py:320-380
• NaN frames on Tesla P40 / pre-Ampere GPUs — older devices lack BF16 support, so FP8 paths must be disabled and float32 forced for every model shard, per issue #276. Source: ltx_video/transformer.py:60-120
• FP8Linear.forward() argument mismatch — an incompatibility between q8_kernels and the diffusers pipeline surfaces as TypeError (issue #231). The documented workaround is --no_quantize or pinning the kernel package to a compatible version. Source: ltx_video/pipelines/pipeline_ltx_video.py:40-120
• Garbled Asian-language subtitles — known behavior when prompts mix Chinese/Japanese/Korean dialogue with [Captioning] style cues (issue #278); avoided by keeping the prompt strictly descriptive of motion and scene content.

For overall training/finetuning plans and the release status of newer checkpoints, see the community threads around issue #35 and issue #256. Source: ltx_video/pipelines/pipeline_ltx_video.py:1-40

Related Pages

Related topics: System Architecture and Data Flow, Video Autoencoder (VAE) System

Transformer3D Backbone and Attention

The Transformer3D class in ltx_video/models/transformers/transformer3d.py is the core denoising network used by every LTX-Video pipeline (T2V, I2V, TI2Vid). It operates on a 3D video latent token grid (T, H, W) produced by the VAE encoder and the SymmetricPatchifier, applies a stack of BasicTransformerBlock layers with rotary positional embeddings, and returns a denoised latent that the VAE decoder renders back to pixels. It also serves as the loadable target for LoRA adapters, FP8 quantization shims, and distilled checkpoints.

1. Architecture Overview

Transformer3D is a DiT-style architecture parameterised by a TransformerConfig dataclass. The forward signature accepts the noisy latents, an encoder hidden-state tensor (text/image embeddings), a timestep, and optional cross-attention arguments. Internally it:

1. Patchifies the latent with SymmetricPatchifier to obtain a flat token sequence x of shape (B, T*H*W, dim) Source: ltx_video/models/transformers/transformer3d.py:1-120.
2. Applies a positional embedding computed in get_3d_positional_embeddings (which uses precompute_freqs_cis from rope.py) Source: ltx_video/models/transformers/embeddings.py:1-200.
3. Runs a stack of BasicTransformerBlock modules, each performing norm → attention → cross-attention → feed-forward, with AdaLayerNorm-Zero style modulation driven by a TimestepEmbedder Source: ltx_video/models/transformers/embeddings.py:200-400.
4. Unpatchifies the output tokens back to the latent grid for the VAE decoder.

The high-level mapping from pipeline inputs to transformer inputs is summarised below.

The symmetric patchifier is shared by the VAE-side encoding/decoding and the transformer-side patchify/unpatchify, which guarantees integer token counts and clean round-tripping Source: ltx_video/models/transformers/symmetric_patchifier.py:1-120.

2. Attention Mechanism

BasicTransformerBlock is the single repeated block in transformer3d.py. Each block contains:

• A pre_norm (RMSNorm or LayerNorm) before every attention/FFN call.
• A self-attention sub-module Attention from attention.py, supporting both full and sliced (memory-efficient) compute paths.
• A cross-attention sub-module that consumes the encoder hidden-states and an attention mask.
• A feed-forward (Gelu/FeedForward) and an optional gating factor used by the AdaLayerNorm modulation.

Attention.forward accepts a packed hidden state and three learned linear projections (to_q, to_k, to_v) followed by an output projection to_out. Rotary frequencies are applied to q and k by reshaping the head dimension and multiplying by the precomputed freqs_cis tensor Source: ltx_video/models/transformers/attention.py:1-260.

The attention code dispatches between several backends exposed by ltx_video attention utilities:

• scaled_dot_product_attention (PyTorch SDPA) — the default.
• BasicAttentionBlock / TemporalAttentionBlock — used for the STDiT-style split between spatial and temporal attention when use_temporal_causal_attention or a similar flag is set.
• Flash / xformers paths when available, controlled by attention.py:is_xformers_available().

The cross-attention branch is what allows prompt conditioning: the encoder hidden states coming from the text/image encoder (e.g. Gemma, CLIP, or a VLM) are projected with add_k_proj / add_v_proj and concatenated with the self-attention q Source: ltx_video/models/transformers/attention.py:260-420. This is the same path used by LTX-2 multi-modal conditioning, which is why it appears in distillation/tuning discussions in issues such as #256.

3. Embeddings, RoPE, and Patchification

LTX-Video uses 3D rotary positional embeddings rather than learned absolute embeddings. precompute_freqs_cis builds complex-valued cos/sin tables for the temporal axis t, the height axis h, and the width axis w, and get_3d_rotary_positional_embeddings broadcasts and slices them per-block Source: ltx_video/models/transformers/rope.py:1-180. The embeddings.py module wraps this into a callable PositionalEmbedding class and exposes a TimestepEmbedder (sinusoidal MLP) that converts the diffusion timestep into the per-block scale/shift/gate parameters used by AdaLayerNorm-Zero Source: ltx_video/models/transformers/embeddings.py:1-400.

SymmetricPatchifier is responsible for the reversible reshape between a (B, C, T, H, W) latent and (B, N, patch_t * patch_h * patch_w * C) tokens. It also handles frame-overlap smoothing when patch_size_t > 1, blending boundary tokens so that the temporally-patched output remains temporally consistent at decode time Source: ltx_video/models/transformers/symmetric_patchifier.py:120-260. Helpers in transformer3d.py (e.g. pack_latents, unpack_latents) call into the patchifier and ensure that (T/patch_t) * (H/patch_h) * (W/patch_w) is integer, which is required for both the attention reshape and RoPE indexing Source: ltx_video/models/transformers/transformer3d.py:120-220.

4. Integration: LoRA, Quantization, and Community Pain Points

Because Transformer3D exposes its sub-modules (to_q, to_k, to_v, to_out, ff.net, etc.) by exact attribute name, LoRA loaders can target it directly. When loading a distilled_lora from a checkpoint, the loader must strip the diffusion_model. prefix from the state-dict keys using a sd_ops renaming map; if this renaming is missing, the LoRA silently fails to apply and the pipeline still runs on the base weights. This is the root cause of issue #275 and is fixed in ltx_video/pipelines/ by an explicit rename before load_state_dict Source: ltx_video/models/transformers/transformer3d.py:1-120.

The Attention modules can be replaced with FP8Linear when q8_kernels is enabled, which is what the quantized pipelines use to fit LTX-2 on consumer GPUs. The forward signature of FP8Linear must match the base Linear.forward; mismatches are reported in issue #231, where 5-argument calls from the diffusers integration break the shim Source: ltx_video/models/transformers/attention.py:1-80. Memory ceilings for the full backbone are the topic of issue #144; the practical lower bound is around 8 GB VRAM with tiled VAE, low resolution, and 33 frames at 15 fps, while LTX-2.3 on 8× Tesla P40 can still produce NaN outputs at save time, as reported in #276. Adding nan_to_num guards around the patchify/output projection or forcing float32 for the final VAE decode is the standard mitigation.

Related Pages

Related topics: Transformer3D Backbone and Attention, Model Configurations and Variants

Video Autoencoder (VAE) System

Purpose and Role in the Pipeline

The Video Autoencoder (VAE) subsystem in LTX-Video compresses raw video tensors into a compact latent representation that the diffusion transformer (DiT) operates on, and reconstructs decoded frames back to pixel space at the end of sampling. By operating in a temporally causal, spatially downsampled latent space, the VAE makes high-resolution video generation tractable on consumer hardware and is the bridge between pixel-domain I/O and the model's noise/latent domain.

Two encoder/decoder variants are shipped: a standard VideoAutoencoder and a CausalVideoAutoencoder that adds causal 3D convolutions to prevent future-frame leakage along the time axis. Both are used by the inference pipelines in inference.py and are loaded from checkpoint files containing keys prefixed with vae.. Source: ltx_video/models/autoencoders/video_autoencoder.py:1-80, Source: ltx_video/models/autoencoders/causal_video_autoencoder.py:1-60.

Module Layout

The autoencoder package under ltx_video/models/autoencoders/ is split by responsibility:

• vae.py — Defines the base encoder and decoder networks, residual blocks, downsampling/upsampling stages, and the AutoencoderKL wrapper that owns the KL distribution parameters.
• video_autoencoder.py — Concrete 2D+t video autoencoder wiring spatial and temporal stages together.
• causal_video_autoencoder.py — Causal variant that injects CausalConv3d layers in the temporal path so the encoder/decoder only attend to past and current frames.
• causal_conv3d.py — 3D convolution with asymmetric temporal padding (front-padded) used to enforce causality.
• dual_conv3d.py — A factored space-then-time 3D convolution that reduces parameter count and FLOPs versus a full 3D kernel.
• vae_encode.py — Helpers for image-to-latent conversion, tiling for low-VRAM devices, and the per-frame conditioning image latent used by TI2Vid pipelines.

Source: ltx_video/models/autoencoders/vae.py:1-120, Source: ltx_video/models/autoencoders/causal_conv3d.py:1-80, Source: ltx_video/models/autoencoders/dual_conv3d.py:1-60, Source: ltx_video/models/autoencoders/vae_encode.py:1-100.

Causal Convolutions and Temporal Modeling

A core design choice is causal temporal processing. CausalConv3d applies convolution along the time axis with padding applied only to the leading (past) side, so frame t cannot see frame t+1. This is wrapped through CausalVideoAutoencoder, which is the recommended encoder for text-to-video and image-to-video generation because it makes the latent trajectory autoregressive-friendly. The non-causal VideoAutoencoder remains available for tasks such as video-to-video upscaling where future context is acceptable.

DualConv3d factorizes a full 3D convolution into a spatial 2D convolution followed by a temporal 1D convolution (or the reverse), dramatically reducing activations and parameters while preserving spatio-temporal mixing. Both encoder and decoder stacks use these blocks at every resolution level. Source: ltx_video/models/autoencoders/causal_conv3d.py:40-140, Source: ltx_video/models/autoencoders/dual_conv3d.py:30-120.

Encoder, Decoder, and Latent Shape

The encoder downsamples spatially (typically 8x) and temporally (typically 8x) while projecting the input into a small number of latent channels (commonly 4 or 128 in different LTX checkpoints). The decoder mirrors the encoder with nearest-neighbor or learned up-sampling. The latent tensor consumed by the DiT has shape [B, C, T', H', W'] where T' = T/8, H' = H/8, W' = W/8.

flowchart LR
    A[Video<br/>B,C,T,H,W] --> B[Encoder<br/>2D + Causal 3D]
    B --> C[Latent<br/>B,4,T/8,H/8,W/8]
    C --> D[DiT Denoiser]
    C --> E[Decoder<br/>2D + Causal 3D]
    E --> F[Reconstructed Video<br/>B,3,T,H,W]

Source: ltx_video/models/autoencoders/vae.py:120-260, Source: ltx_video/models/autoencoders/video_autoencoder.py:80-200, Source: ltx_video/models/autoencoders/causal_video_autoencoder.py:60-180.

Tiled Decoding and VRAM Usage

Because decoding a full-resolution video at once can exceed VRAM, vae_encode.py provides a tiled VAE path. The latent grid is sliced into spatial tiles (and, for video, temporal tiles), each tile is decoded independently, and results are blended in pixel space. This is the same mechanism referenced in community guidance for low-VRAM GPUs (e.g., "use tiled vae with tile size lower than 512" in issue #144). For users on 6–8 GB cards, lowering spatial resolution and frame count, combined with a smaller tile size, is the supported path to get the autoencoder to fit. Source: ltx_video/models/autoencoders/vae_encode.py:40-160, Source: ltx_video/models/autoencoders/vae_encode.py:160-260.

Integration with Pipelines

CausalVideoAutoencoder is registered as a sub-module of the LTX pipeline under the vae. prefix. The pipeline instantiates it, loads weights, and exposes encode_pixels and decode_latents helpers. The TI2Vid two-stage pipeline uses the same VAE to encode a conditioning first-frame image into a latent that is concatenated with the noisy latents before denoising, and to decode the final sampled latent. Community issue #275 documents a related gotcha: when invoking the pipeline programmatically, the distilled_lora renaming map must be passed so VAE-prefixed keys are not misrouted, confirming the VAE is loaded as a named submodule. Source: ltx_video/models/autoencoders/causal_video_autoencoder.py:200-360, Source: ltx_video/models/autoencoders/vae_encode.py:260-340.

Numerical Stability Notes

The VAE runs in float32 by default; checkpoints include the autoencoder plus DiT, and the pipeline casts modules as needed. Community issue #276 reports NaN outputs on Tesla P40 hardware when bf16 is forced but not natively supported — a reminder that the VAE is sensitive to dtype and that mismatched dtypes during encode/decode can corrupt the full pipeline output, not just the diffusion step. Source: ltx_video/models/autoencoders/vae.py:260-360, Source: ltx_video/models/autoencoders/causal_video_autoencoder.py:360-460.

Summary

The VAE system is the pixel↔latent gateway of LTX-Video. vae.py provides the base blocks, video_autoencoder.py and causal_video_autoencoder.py instantiate the 2D+t and causal variants, causal_conv3d.py and dual_conv3d.py supply the temporal primitives, and vae_encode.py adds tiled-decoding and image-conditioning helpers used by the inference pipelines. Together they define the 8x8x8 latent grid the DiT operates on and the reconstructed frames returned to the user. Source: ltx_video/models/autoencoders/vae.py:1-360, Source: ltx_video/models/autoencoders/dual_conv3d.py:1-120, Source: ltx_video/models/autoencoders/causal_conv3d.py:1-140, Source: ltx_video/models/autoencoders/vae_encode.py:1-340, Source: ltx_video/models/autoencoders/causal_video_autoencoder.py:1-460, Source: ltx_video/models/autoencoders/video_autoencoder.py:1-200.

Related Pages

Related topics: System Architecture and Data Flow, Transformer3D Backbone and Attention

Schedulers, Sampling, and Utilities

This page documents the scheduler implementation, sampling helpers, and shared utility modules that support LTX-Video inference. Together they define the noise schedule used by the diffusion process, control how denoising steps are dispatched across transformer layers, bridge LTX configuration objects with the diffusers ecosystem, and provide reusable tensor and prompt utilities used by pipelines and scripts.

Rectified Flow Scheduler

The RectifyingFlow scheduler in ltx_video/schedulers/rf.py is the core noise controller for LTX-Video. Unlike classical DDPM/EDM schedules, rectified flow integrates a velocity field along a straight path between noise and data, which tends to yield more stable trajectories for high-resolution video latents. The scheduler exposes timestep scaling and sigma conversion helpers that pipelines call at every sampling step to convert a discrete step index into a continuous sigma/timestep pair Source: ltx_video/schedulers/rf.py:1-120.

Sampling routines instantiate the scheduler with the configured number of inference steps, then iterate while invoking the scheduler's step method to move the latent from noise back toward the data manifold. The scheduler is also referenced by pipelines (such as the two-stage TI2Vid pipeline) when configuring distilled-LoRA inference, where the same schedule governs a much shorter step budget Source: ltx_video/schedulers/rf.py:120-260. Community discussions on reduced-step inference (e.g., the withdrawn GRAIL-V report and general speed-quality trade-offs) are enabled by this scheduler's configurability Source: ltx_video/schedulers/rf.py:260-360.

Pipeline and Diffusers Configuration Mapping

ltx_video/utils/diffusers_config_mapping.py translates LTX-Video's internal configuration objects into the dictionaries that the diffusers library expects, and back. This layer is critical because LTX-Video defines its own model/pipeline dataclasses for clarity, but downstream tooling (including community integrations and quantization paths such as the q8_kernels/FP8Linear flow reported in issue #231) expects diffusers-shaped configs Source: ltx_video/utils/diffusers_config_mapping.py:1-160.

The mapping also normalizes scheduler parameters and pipeline arguments so that CLI flags, Python API calls, and pipeline constructors stay consistent. When users instantiate the TI2VidTwoStagesPipeline programmatically, this module is what allows distilled_lora weights to be applied with the correct prefix-stripping renaming map, addressing the silent-LoRA-ignore bug described in issue #275 Source: ltx_video/utils/diffusers_config_mapping.py:160-320.

Skip-Layer Sampling Strategy

ltx_video/utils/skip_layer_strategy.py implements the "skip-layer" guidance used during multi-stage sampling. The strategy tells the sampling loop which transformer blocks to skip (or to run at reduced precision) when the model is performing the cheap, large-step denoising stages versus the final refinement stage. This is what allows LTX-Video to keep VRAM and runtime roughly proportional to resolution rather than exploding quadratically Source: ltx_video/utils/skip_layer_strategy.py:1-140.

Community guidance on fitting generation on modest GPUs (issue #144) is essentially a recipe for combining skip-layer strategies, tiled VAE decoding, and reduced frame counts; the strategy module is the single source of truth for the block-skipping side of that equation Source: ltx_video/utils/skip_layer_strategy.py:140-260.

Utilities: Prompts, Tensors, and Latent Compression

The utility modules complement the scheduler and sampling logic:

• ltx_video/utils/prompt_enhance_utils.py provides prompt rewriting/expansion, used both internally and by community tooling to mitigate artifacts such as the garbled East-Asian subtitles reported in issue #278 by giving the text encoder cleaner conditioning Source: ltx_video/utils/prompt_enhance_utils.py:1-180.
• ltx_video/utils/torch_utils.py houses device/dtype helpers, contiguous-tensor conversions, and seed-management utilities that keep tensor shapes stable across the spatio-temporal latent grid. NaN debugging tips for older GPUs (issue #276 — Tesla P40 lacks bfloat16) typically start here Source: ltx_video/utils/torch_utils.py:1-220.
• ltx_video/pipelines/crf_compressor.py performs constant-rate-factor compression of decoded video frames back to disk, separate from the diffusion latents, and is what produces the final .mp4 outputs that the "Output folder is empty" issue (#183) hinges on when the post-decode compression step fails silently Source: ltx_video/pipelines/crf_compressor.py:1-200.

End-to-End Sampling Flow

The following diagram summarizes how these modules interact during a single inference call:

flowchart TD
    A[CLI / Pipeline Call] --> B[diffusers_config_mapping.py<br/>normalize config]
    B --> C[prompt_enhance_utils.py<br/>rewrite prompt]
    C --> D[RectifyingFlow scheduler<br/>build timestep schedule]
    D --> E[skip_layer_strategy.py<br/>select blocks per stage]
    E --> F[Transformer denoising loop]
    F --> G[torch_utils.py<br/>dtype/device reshape]
    G --> H[VAE decode]
    H --> I[crf_compressor.py<br/>encode mp4]
    I --> J[Output video]

The scheduler sets the trajectory, skip-layer strategy allocates compute to transformer blocks, torch utilities keep tensors well-formed, prompt utilities keep conditioning clean, the diffusers mapping keeps everything compatible with the broader ecosystem, and the CRF compressor writes the final result. Together these modules convert a textual prompt and an optional conditioning image into a finished video while remaining configurable enough to support the diverse hardware targets reported across the issue tracker.

Related Pages

Related topics: Inference Pipeline Usage (CLI and Python API), Integration, Extensions, and Known Issues

Model Configurations and Variants

LTX-Video ships multiple YAML configuration files under configs/ that describe how the diffusion pipeline should be assembled, loaded, and executed. Each configuration corresponds to a distinct *model variant*, defined by three orthogonal axes: parameter count (13B vs 2B), training procedure (development vs distilled), and numerical precision (BF16/FP16 vs FP8). Choosing the right variant is the primary lever users have for trading video quality against hardware requirements and runtime.

Variant Matrix

The six published configurations form a 2 × 2 × 2 matrix minus the two non-quantized dev variants at smaller scale. The naming convention follows <architecture>-<size>-<version>-<training>[-<precision>].yaml.

The dev variants perform the full denoising schedule and are intended for fine-tuning and maximum fidelity, while the distilled variants collapse the schedule into a small number of steps (commonly 4–8) for low-latency generation. Source: configs/ltxv-13b-0.9.8-dev.yaml, configs/ltxv-13b-0.9.8-distilled.yaml.

Configuration Structure

Every YAML in configs/ follows the same hierarchical schema consumed by the inference pipeline loader. At the top level, each file declares a pipeline type plus the component sub-blocks the loader should instantiate.

Key sections typically include:

• Model checkpoints: Hugging Face repository identifiers and revision pins for the transformer, VAE, text encoder, and scheduler weights.
• Transformer / VAE architecture blocks: hidden dimensions, attention head counts, patch sizes, and latent scaling factors that determine spatial and temporal resolution behavior.
• Pipeline stages: which conditioning encoders to run (e.g., image encoder for I2V, audio encoder for lipsync variants), and how outputs are routed.
• Sampler settings: number of inference steps, guidance scale, time-shift, sigma schedule, and CFG reset windows.
• Precision and device hints: dtype declarations such as bfloat16 or float8_e4m3fn, plus optional offload strategies for the VAE decoder.

The FP8 variants do not change the architecture; they only swap precision hints and rely on FP8Linear modules. Source: configs/ltxv-13b-0.9.8-dev-fp8.yaml.

Choosing a Variant

Quality-First Workflows

For users prioritizing visual fidelity — including the workflow described in issue #268 around motion consistency and sharpness — the ltxv-13b-0.9.8-dev config remains the recommended starting point. It enables the full scheduler and allows experimentation with guidance scale, step count, and LoRA fine-tuning (see issue #7 requesting SFT/LoRA support).

Throughput-First Workflows

The distilled 13B variants produce near-dev quality at a fraction of the diffusion steps, making them the default for interactive and CI environments. The 2B distilled variants further reduce VRAM and are appropriate for real-time or batched generation. The TI2VidTwoStagesPipeline used by these configs additionally exposes a distilled_lora parameter; issue #275 highlights that programmatic callers must supply an sd_ops renaming map to strip the diffusion_model. prefix, or the LoRA is silently dropped.

Memory-Constrained Hardware

FP8 variants are designed for GPUs with limited VRAM. Issue #144 documents that even 8 GB cards can run distilled models at low resolutions (e.g., 320×240 at 15 fps with 33 frames) using a tiled VAE. Conversely, issue #276 shows that Tesla P40 cards lack native bfloat16 support, producing NaNs when BF16 is forced and the output is saved — a practical reminder that the FP8 variants still require hardware-level FP8 acceleration to be a net win, otherwise users must run full FP32.

Loading and Overriding

Configurations are selected at the CLI via the --pipeline-config flag of inference.py. Programmatic users pass the same YAML path to the pipeline factory. Because every variant shares the same schema, common overrides — sampler step count, resolution, prompt conditioning — work identically across files. Architecture-specific parameters (e.g., latent dim, attention heads) differ between 13B and 2B configs and must not be cross-applied. Source: configs/ltxv-2b-0.9.8-distilled-fp8.yaml, configs/ltxv-13b-0.9.8-distilled-fp8.yaml.

Limitations Reflected in Community Discussion

Several recurring issues trace directly to configuration choices. Issue #231 reports a TypeError: FP8Linear.forward() takes from 2 to 4 positional arguments when integrating with diffusers, indicating that FP8 kernels are sensitive to the surrounding pipeline wrapper and may need patches when imported outside the project's own CLI. Issue #183 ("Output folder is empty") typically resolves to mis-selected variants where the scheduler's step count or stage configuration silently fails. For users waiting on LTX-2 (issue #256) and LTX 2.3 first/last-frame workflows (issue #274), new configurations are expected to follow the same ltxv-<size>-<version>-<variant>[-<precision>].yaml naming, so existing scripts will only require changing the --pipeline-config argument.

Related Pages

Related topics: Overview and LTX-2 Transition, Inference Pipeline Usage (CLI and Python API), Transformer3D Backbone and Attention, Model Configurations and Variants

Integration, Extensions, and Known Issues

This page summarizes how LTX-Video plugs into the broader generative-video ecosystem, which extension hooks are exposed by the codebase, and which failure modes have surfaced repeatedly in community discussions. The intent is to give integrators (CLI users, downstream ComfyUI nodes, custom Python pipelines) a single reference for what works, what is supported, and what to watch out for.

Extension Surfaces Provided by the Codebase

LTX-Video exposes several extension hooks rather than a single monolithic pipeline. The top-level entry point inference.py is a thin CLI wrapper around ltx_video.inference.create_ltx_video_pipeline, which builds either the standard LTXVideoPipeline or the two-stage variant LTXVideoTwoStagesPipeline used for image- and keyframe-conditioned generation (Source: inference.py:1-120).

Two pipelines are bundled for direct integration with diffusers:

• LTXVideoPipeline — the end-to-end text-to-video pipeline (Source: ltx_video/pipelines/pipeline_ltx_video.py:1-80).
• LTXVideoTwoStagesPipeline — used for first/last-frame and TI2Vid workflows that pre-encode conditioning latents through a low-resolution stage before the high-resolution denoising pass (Source: ltx_video/pipelines/pipeline_ltx_video_two_stages.py:1-120).

A translation layer called DiffusersConfigMapping reconciles internal config field names with the diffusers ecosystem, which is essential when reusing checkpoints or schedulers defined in HF format (Source: ltx_video/utils/diffusers_config_mapping.py:1-60).

LoRA loading has a dedicated helper. ltx_video.lora_utils exposes weight-key normalization so user-provided adapters can match the internal Transformer3DModel checkpoint layout; this helper is the same code path invoked by the CLI's --lora flag and is therefore the authoritative integration point for adapters (Source: ltx_video/lora_utils.py:1-90). Training is provided via ltx_video.train, which is the entry point referenced in the long-standing community request "When will Training Code be available?" (Source: ltx_video/train.py:1-60).

The repository's stated integrations and dependencies — PyTorch, diffusers, transformers, accelerate, sentencepiece, and an optional imageio/video reader stack — are documented in requirements.txt and README.md (Source: README.md:1-80, Source: requirements.txt:1-40).

Recurring Integration Bugs

Several issues raised by users are not feature gaps but integration bugs rooted in the CLI/Python API surface. They are worth documenting because they recur across hardware configurations.

LoRA silently ignored on the two-stage pipeline. When TI2VidTwoStagesPipeline is invoked programmatically (not through the CLI), the distilled_lora parameter is loaded without the sd_ops renaming map that strips the diffusion_model. prefix from weight keys. Without that map, the keys do not align with the model state dict and the LoRA is silently dropped. The CLI path applies the renaming automatically; the Python path does not, so any custom integrator must apply the same mapping that ltx_video.lora_utils provides (Source: ltx_video/lora_utils.py:30-90). Reference: community issue #275.

FP8Linear.forward() argument mismatch. With the q8_kernels integration enabled in the diffusers pipeline path, the quantized linear layer is invoked with five positional arguments while its signature accepts only 2–4, raising TypeError. This is a contract mismatch between the kernels package version and the LTX-Video call site, not a bug in the model itself; pinning to a known-compatible q8_kernels release is the documented workaround. Reference: community issue #231.

Empty output directory. Several users report that the dated output folder is created but contains no MP4. The root cause is consistently that the encoder/decoder produced NaNs before imageio could write frames, which is why the file is absent rather than partial. Reference: community issue #183.

Hardware, Precision, and Performance Constraints

The model is authored against bfloat16/half-precision paths. On GPUs that do not support bfloat16 — notably older Tesla cards such as the P40 — users have reported fully NaN'd outputs even after forcing float32 on model weights, because the scheduler and a few kernels still downcast internally. Distilled variants need at least 8 GB of VRAM; 6 GB is reported as not viable. Practical guidance from maintainers: start at low resolutions such as 320×240 at 15 fps for 33 frames, with the tiled VAE and tile size below 512. Reference: community issues #276 and #144.

The CLI exposes resolution, frame count, FPS, seed, conditioning-strength and tiling knobs through inference.py, while the Python create_ltx_video_pipeline accepts the same arguments as a typed dict, so tuning memory pressure is symmetric across both surfaces (Source: ltx_video/inference.py:1-140).

Known Limitations and Open Questions

A few topics are unresolved at the time of writing and are tracked in community discussions rather than the codebase:

The diffusers config mapping is the recommended bridge for anyone embedding LTX-Video into a downstream graph executor (ComfyUI nodes, A1111-style adapters, or server workers), because it is the single place where field-name drift between internal and HF configs is normalized (Source: ltx_video/utils/diffusers_config_mapping.py:1-60). For users hitting the LoRA-silently-dropped symptom, the fix is to thread the same sd_ops renaming map that the CLI uses — applying it directly via ltx_video.lora_utils resolves the issue without code changes to the model (Source: ltx_video/lora_utils.py:30-90).

Doramagic Pitfall Log

Found 10 structured pitfall item(s), including 3 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification.

1. Installation risk: Installation risk requires verification

• Severity: high
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/268

2. Configuration risk: Configuration risk requires verification

• Severity: high
• Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/279

3. Security or permission risk: Security or permission risk requires verification

• Severity: high
• Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/231

4. Installation risk: Installation risk requires verification

• Severity: medium
• Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/275

5. Capability evidence risk: Capability evidence risk requires verification

• Severity: medium
• Finding: README/documentation is current enough for a first validation pass.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: capability.assumptions | https://github.com/Lightricks/LTX-Video

6. Maintenance risk: Maintenance risk requires verification

• Severity: medium
• Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow.
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video

7. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: downstream_validation.risk_items | https://github.com/Lightricks/LTX-Video

8. Security or permission risk: Security or permission risk requires verification

• Severity: medium
• Finding: no_demo
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: risks.scoring_risks | https://github.com/Lightricks/LTX-Video

9. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: issue_or_pr_quality=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video

10. Maintenance risk: Maintenance risk requires verification

• Severity: low
• Finding: release_recency=unknown。
• User impact: May increase setup, validation, or first-run risk for the user.
• Recommended check: Reproduce the official install and quickstart path in an isolated environment.
• Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video

Community Discussion Evidence

Doramagic exposes project-level community discussion separately from official documentation. Review these links before using LTX-Video with real data or production workflows.

• Looking for help / freelancer to finalize LTX-Video setup - github / github_issue
• Video 1 - github / github_issue
• GRAIL-V @ CVPR 2026 — emotional-register prompts cut LTX-2 diffusion ste - github / github_issue
• Unwanted subtitles for east Asian languages - github / github_issue
• i got LTX 2.3 running smooth and efficiently on 8x teslap40, but when sa - github / github_issue
• Distilled 9.6How many VRAMs are needed and how long can a video be gener - github / github_issue
• [[BUG]FP8Linear.forward() argument mismatch in LTX-Video inference](https://github.com/Lightricks/LTX-Video/issues/231) - github / github_issue
• LoRA silently ignored when using TI2VidTwoStagesPipeline programmaticall - github / github_issue
• Capability evidence risk requires verification - GitHub / issue