# https://github.com/Lightricks/LTX-Video Project Manual Generated at: 2026-07-05 17:15:40 UTC ## Table of Contents - [Overview and LTX-2 Transition](#page-1) - [System Architecture and Data Flow](#page-2) - [Inference Pipeline Usage (CLI and Python API)](#page-3) - [Transformer3D Backbone and Attention](#page-4) - [Video Autoencoder (VAE) System](#page-5) - [Schedulers, Sampling, and Utilities](#page-6) - [Model Configurations and Variants](#page-7) - [Integration, Extensions, and Known Issues](#page-8) ## Overview and LTX-2 Transition ### Related Pages Related topics: [System Architecture and Data Flow](#page-2), [Model Configurations and Variants](#page-7)
Related Source Files The following source files were used to generate this page: - [README.md](https://github.com/Lightricks/LTX-Video/blob/main/README.md) - [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py) - [ltx_video/pipelines/__init__.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/__init__.py) - [ltx_video/pipelines/ti2vid_two_stages_pipeline.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/ti2vid_two_stages_pipeline.py) - [ltx_video/models/transformers/transformer3d.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/transformer3d.py) - [ltx_video/models/autoencoders/vae.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/vae.py) - [ltx_video/utils/quantization.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/quantization.py) - [requirements.txt](https://github.com/Lightricks/LTX-Video/blob/main/requirements.txt) - [configs/ltxv-13b-0.9.7-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.7-dev.yaml) - [docs/LTX-2.md](https://github.com/Lightricks/LTX-Video/blob/main/docs/LTX-2.md)
# Overview and LTX-2 Transition ## Purpose and Scope LTX-Video is an open-source video diffusion model maintained by Lightricks. It provides a unified pipeline stack for text-to-video (T2V) and image-to-video (I2V) generation, a Transformer-3D backbone, a temporal VAE, and an inference entry point that doubles as a CLI and a programmatic API. Source: [README.md:1-60](). The repository currently spans two major generations on the same codebase: - **LTX-Video (legacy line)** — checkpoint series such as `ltxv-13b-0.9.7-dev` and the "distilled 9.6" line, exposed via `inference.py` and the per-config YAMLs in `configs/`. - **LTX-2** — the next-generation family that introduces an audio branch, longer context windows, and a rewritten two-stage sampler; the codebase is the migration surface for users moving from LTX-Video to LTX-2. Source: [docs/LTX-2.md:1-40](). The "Overview and LTX-2 Transition" page therefore documents what the project *is* today (the legacy line, which most open-source artifacts still ship) and what the LTX-2 migration *changes* (config keys, pipeline classes, weight loading, hardware expectations). ## High-Level Architecture The system is organised around a small number of composable building blocks: | Component | Path | Role | |---|---|---| | Inference entry | `inference.py` | CLI + Python API that loads a YAML config, instantiates a pipeline, and writes frames to disk. | | Pipelines | `ltx_video/pipelines/` | Stage orchestrators: `TI2VidTwoStagesPipeline` is the primary two-stage image+text-to-video pipeline. | | Backbone | `ltx_video/models/transformers/transformer3d.py` | The 3D Transformer that performs denoising across space and time. | | VAE | `ltx_video/models/autoencoders/vae.py` | Temporal autoencoder used for latent encoding/decoding and tiled decoding. | | Quantization | `ltx_video/utils/quantization.py` | Wraps `FP8Linear` (q8_kernels) and the bfloat16/float32 fallback paths. | Source: [inference.py:1-120](), [ltx_video/pipelines/__init__.py:1-30](), [ltx_video/utils/quantization.py:1-80](). ```mermaid flowchart LR A[Prompt / Start image] --> B[inference.py] B --> C[Pipeline class] C --> D[Transformer3D denoiser] C --> E[Temporal VAE] D --> F[Latents] F --> E E --> G[Output MP4 frames] C -.quantized via.-> H[FP8Linear / q8_kernels] ``` ## Pipeline Inventory and the Two-Stage Sampler `TI2VidTwoStagesPipeline` is the workhorse for I2V generation. It first runs a "keyframe" denoise stage conditioned on the start frame, then a second "interpolation" stage that fills in the remaining frames at full temporal resolution. This split is what enables long, smooth clips at moderate VRAM. Source: [ltx_video/pipelines/ti2vid_two_stages_pipeline.py:1-90](). When invoking the pipeline programmatically rather than through the CLI, callers must pass a `distilled_lora` together with an `sd_ops` renaming map so that LoRA weight keys prefixed with `diffusion_model.` are mapped back to the pipeline's expected names. Without this map the LoRA loads silently and has no effect — a known footgun flagged by the community. Source: [ltx_video/pipelines/ti2vid_two_stages_pipeline.py:120-180](), community report in issue #275. ## The LTX-2 Transition LTX-2 keeps the same monorepo but tightens several contracts that downstream users must respect: 1. **Config schema.** Legacy `ltxv-13b-0.9.7-dev` YAMLs are still loadable, but LTX-2 introduces new keys (`audio_*`, longer `temporal_context`) that legacy pipelines ignore. Source: [configs/ltxv-13b-0.9.7-dev.yaml:1-60](), [docs/LTX-2.md:40-120](). 2. **Weight naming.** Checkpoint prefixes changed; the renaming map used by `TI2VidTwoStagesPipeline` for LoRA loading is also the recommended translation layer when porting custom weights to LTX-2. Source: [ltx_video/pipelines/ti2vid_two_stages_pipeline.py:120-180](). 3. **Diffusers integration.** LTX-2 is the first generation in the repo that targets the upstream `diffusers` pipeline interface end-to-end, which is what surfaces issues such as the `FP8Linear.forward()` arity mismatch in the q8_kernels integration path. Source: [ltx_video/utils/quantization.py:1-80](), issue #231. 4. **Hardware expectations.** LTX-2's longer context pushes VRAM upward; the community guidance for the legacy distilled 9.6 line (≈8 GB at 320×240, 33 frames, tiled VAE) is a lower bound, not a recommendation for LTX-2. Source: issue #144. ## Known Limitations Surfaced by the Community Several open issues map directly to transitional friction: - **NaN outputs on Tesla P40** when saving the video — P40 lacks bfloat16, so users forced to float32 still hit NaN on the writer path with LTX 2.3. Source: issue #276. - **East-Asian language subtitles** are produced almost deterministically (≈99%) even when no subtitle is requested, both for pure T2V and for lip-sync workflows. Source: issue #278. - **FP8 arity bug** (`FP8Linear.forward() takes from 2 to 4 positional arguments`) when running LTX-Video inference through certain `diffusers` builds — fixed on the LTX-2 branch. Source: issue #231. - **Empty output folders** ("Output folder is empty", issue #183) trace back to CLI invocation mismatches and are amplified during the LTX-2 transition because new config keys default differently. - **I2V "first frame / last frame"** workflows (issue #274) and fine-tuning / LoRA training support (issue #35, issue #7) are the two most-requested capabilities that LTX-2 is expected to stabilise. For users migrating, the safe path is: pin a known-good legacy config, audit LoRA loaders for the `sd_ops` map, validate FP8 quant on the target diffusers commit, and only then enable LTX-2-specific config keys. Source: [README.md:60-140](), [docs/LTX-2.md:1-200](). --- ## System Architecture and Data Flow ### Related Pages Related topics: [Inference Pipeline Usage (CLI and Python API)](#page-3), [Transformer3D Backbone and Attention](#page-4), [Video Autoencoder (VAE) System](#page-5)
Related Source Files The following source files were used to generate this page: - [ltx_video/pipelines/pipeline_ltx_video.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/pipeline_ltx_video.py) - [ltx_video/inference.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/inference.py) - [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py) - [ltx_video/models/transformers/transformer3d.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/transformer3d.py) - [ltx_video/models/autoencoders/vae.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/vae.py) - [ltx_video/schedulers/rf.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/schedulers/rf.py) - [ltx_video/utils/pipeline_utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/pipeline_utils.py) - [ltx_video/lora_loader.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/lora_loader.py) - [ltx_video/quantization/q8_kernels.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/quantization/q8_kernels.py)
# System Architecture and Data Flow LTX-Video is a transformer-based latent video diffusion system. The repository separates orchestration (pipelines and CLI entrypoints), numerical models (transformer, VAE, scheduler, text encoder), and cross-cutting utilities (LoRA loading, FP8 quantization, resolution conditioning). The system takes a text prompt and optional conditioning images, produces a denoised latent video tensor, and decodes it into pixel-space frames. ## High-Level Architecture The runtime is layered so that model components know nothing about CLI flags, while the CLI knows nothing about tensor shapes. - **Entry layer** — `inference.py` at the repository root parses CLI arguments, validates paths, and dispatches to the package-level entrypoint `ltx_video/inference.py`, which constructs the pipeline object and invokes generation. Source: [inference.py:1-120]() and [ltx_video/inference.py:1-200](). - **Pipeline layer** — `ltx_video/pipelines/pipeline_ltx_video.py` defines `LTXVideoPipeline` and the two-stage variant `TI2VidTwoStagesPipeline`. Pipelines own the denoising loop, conditioning assembly, and decoding. Source: [ltx_video/pipelines/pipeline_ltx_video.py:1-400](). - **Model layer** — `ltx_video/models/transformers/transformer3d.py` implements the spatiotemporal transformer, `ltx_video/models/autoencoders/vae.py` provides the causal 3D VAE, and `ltx_video/schedulers/rf.py` supplies the rectified-flow noise schedule. Source: [ltx_video/models/transformers/transformer3d.py:1-300](), [ltx_video/models/autoencoders/vae.py:1-300](), and [ltx_video/schedulers/rf.py:1-150](). ## Pipeline Data Flow Generation follows a fixed sequence that the pipeline orchestrates end-to-end. Latents flow from a low-dimensional noise tensor up to pixel-space frames; conditioning flows in from the side. ```mermaid flowchart LR A[Text Prompt] --> B[Text Encoder] IMG[First/Last Frame] --> ENC[VAE Encoder] B --> COND[Condition Builder] ENC --> COND COND --> DNOISE[Denoising Loop\nLTXVideoPipeline] SCH[RF Scheduler] --> DNOISE X3D[Transformer3D] --> DNOISE DNOISE --> LAT[Latent Video Tensor] LAT --> DEC[VAE Decoder] DEC --> OUT[Pixel Frames] DEC --> SAVE[Video Writer] ``` The pipeline calls the encoder once for images, embeds the prompt, builds the conditioning dict, iterates `num_inference_steps` times calling `Transformer3D.forward`, and finally hands the latent tensor to the VAE decoder. Source: [ltx_video/pipelines/pipeline_ltx_video.py:200-500](). For image-to-video and keyframe-conditioned workflows, `TI2VidTwoStagesPipeline` performs a first coarse stage and a second refining stage that re-injects start/end latents. Source: [ltx_video/pipelines/pipeline_ltx_video.py:400-700](). ## Core Components **Transformer3D** is the denoising network. It consumes a packed latent of shape `(B, C, T, H, W)` together with text embeddings, timestep, and positional embeddings, and returns a noise prediction of the same shape. Source: [ltx_video/models/transformers/transformer3d.py:1-300](). **Causal 3D VAE** encodes images into latent space and decodes latents back into video. The decoder is causal along the time axis so that frame `t` depends only on frames `≤ t`. Tiled decoding is supported for low-VRAM setups; the community report for Distilled 9.6 explicitly recommends `tile size < 512`. Source: [ltx_video/models/autoencoders/vae.py:1-300](). **Rectified-Flow Scheduler** advances the latent by linear interpolation between noise and clean data, parameterized by a small number of inference steps. Distilled checkpoints such as LTX 2.3 use as few as 8 steps. Source: [ltx_video/schedulers/rf.py:1-150](). **Pipeline Utilities** handle resolution snapping, frame-count rounding, and conditioning construction so that dimensions are divisible by the VAE and transformer patch grids. Source: [ltx_video/utils/pipeline_utils.py:1-200](). ## Integration and Optimization Points The architecture exposes three extensibility seams that the community actively exercises. - **LoRA loading** — `ltx_video/lora_loader.py` applies adapter weights at runtime. When invoked programmatically through `TI2VidTwoStagesPipeline`, callers must provide an `sd_ops` renaming map to strip the `diffusion_model.` prefix; without it the LoRA is silently ignored. Source: [ltx_video/lora_loader.py:1-200]() and the report in issue #275. - **FP8 quantization** — `ltx_video/quantization/q8_kernels.py` supplies `FP8Linear`, a drop-in replacement for `nn.Linear`. The kernel expects 2–4 forward arguments; passing extras (as some diffusers versions do) raises a `TypeError` and breaks inference. Source: [ltx_video/quantization/q8_kernels.py:1-200]() and issue #231. - **Numerical stability** — Tesla P40 lacks BF16 support, so users force FP32 and can still encounter NaNs at save time (issue #276). The safe path is to keep the scheduler and VAE in FP32 even when the transformer runs quantized, and to cast the decoded tensor to FP32 before writing. ## Boundary Summary The system is organized so that an integrator can swap the scheduler, replace the VAE for tiled decoding, plug a LoRA into the transformer, or wrap `FP8Linear` around attention projections without touching the pipeline loop. This is what makes the same pipeline serve pure text-to-video, image-to-video with first/last frames, and distilled 8-step variants. Word count: ~870 --- ## Inference Pipeline Usage (CLI and Python API) ### Related Pages Related topics: [Model Configurations and Variants](#page-7), [Integration, Extensions, and Known Issues](#page-8)
Related Source Files The following source files were used to generate this page: - [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py) - [ltx_video/inference.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/inference.py) - [ltx_video/pipelines/__init__.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/__init__.py) - [ltx_video/pipelines/pipeline_ltx_video.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/pipeline_ltx_video.py) - [ltx_video/pipelines/ti2vid_two_stages_pipeline.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/ti2vid_two_stages_pipeline.py) - [ltx_video/scheduler.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/scheduler.py) - [ltx_video/transformer.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/transformer.py) - [ltx_video/utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils.py)
# Inference Pipeline Usage (CLI and Python API) ## Overview The LTX-Video inference pipeline converts text or image prompts into short video clips by orchestrating a video diffusion transformer plus a video VAE decoder. The repository exposes both a CLI entry point (`inference.py` at the repository root) and a Python API (`ltx_video.inference` plus the `ltx_video.pipelines` package) so that the same weights can be reused from shell, notebooks, or downstream tools. The CLI is aimed at one-off generation, while the Python API exposes hooks for two-stage pipelines, LoRA loading, and precision toggles. Source: [inference.py:1-40]() Source: [ltx_video/inference.py:1-30]() ## CLI Usage The root `inference.py` script thinly wraps `ltx_video.inference.main(...)` and surfaces the most common generation arguments as flags: `--prompt`, `--negative_prompt`, `--height`, `--width`, `--num_frames`, `--frame_rate`, `--seed`, `--pipeline_type` (e.g. `text2video`, `image2video`, `ti2vid_two_stages`), `--ckpt_path`, `--output_dir`, and `--pipeline_config`. The script creates a timestamped sub-folder under `output_dir` and writes the MP4 alongside conditioning metadata. Source: [inference.py:42-120]() ```bash python inference.py \ --prompt "A cinematic shot of waves crashing on a rocky shore" \ --height 480 --width 768 --num_frames 97 --frame_rate 24 \ --pipeline_type text2video \ --output_dir ./outputs ``` For image-conditioned generation, supply `--image_path` together with `--pipeline_type image2video`; for keyframe-conditioned two-stage generation, switch to `ti2vid_two_stages`, which is the workflow requested in [issue #274](https://github.com/Lightricks/LTX-Video/issues/274). Source: [ltx_video/inference.py:60-140]() | Flag | Purpose | | --- | --- | | `--pipeline_type` | Selects T2V, I2V, or the two-stage keyframe pipeline | | `--num_inference_steps` | Diffusion steps; lower values for distilled checkpoints | | `--guidance_scale` | CFG strength, typically 1.0–4.5 | | `--tiled_vae` / `--tile_size` | Enables VAE tiling for low-VRAM GPUs | | `--no_quantize` | Disables FP8 quantization (workaround for kernel mismatches) | Source: [ltx_video/inference.py:140-260]() ## Python API Programmatic access lives in `ltx_video.inference` and the `ltx_video.pipelines` package. The exported `main(...)` returns either the constructed pipeline or a result dictionary, making it directly embeddable in notebooks. Source: [ltx_video/inference.py:30-90]() Pipelines are imported from `ltx_video.pipelines`. The standard text/image pipeline lives in `pipeline_ltx_video.py`, while `ti2vid_two_stages_pipeline.py` implements the two-stage image-to-video path that generates a coarse keyframe sequence and then refines it. Source: [ltx_video/pipelines/__init__.py:1-30]() ```python from ltx_video.inference import main as run_inference result = run_inference( prompt="A slow pan across a misty forest at dawn", pipeline_type="text2video", output_dir="./outputs", height=480, width=768, num_frames=65, frame_rate=24, ) ``` When invoking the two-stage pipeline programmatically, a LoRA must be loaded with an `sd_ops` renaming map so that the `diffusion_model.` prefix is stripped from the checkpoint keys; without it the adapter is silently ignored, as reported in [issue #275](https://github.com/Lightricks/LTX-Video/issues/275). Source: [ltx_video/pipelines/ti2vid_two_stages_pipeline.py:80-160]() Utility helpers in `ltx_video.utils.py` cover prompt-embedding bookkeeping and VAE tiling, while `ltx_video.scheduler.py` owns the noise schedule and CFG mixing for both base and distilled checkpoints. Source: [ltx_video/utils.py:1-80]() Source: [ltx_video/scheduler.py:1-60]() ## Architecture, Hardware, and Known Issues ```mermaid flowchart LR A[Prompt / Image] --> B[Tokenizer & Conditioning] B --> C[Latent Diffusion Transformer] C --> D[VAE Decode] D --> E[Temporal Smoothing] E --> F[MP4 Writer] C -. optional LoRA .-> G[Adapter Weights] ``` VRAM requirements scale with `height × width × num_frames`. A distilled 0.9.6 checkpoint at 320×240 with 33 frames fits in roughly 8 GB when the tiled VAE is enabled with a small `--tile_size`; 6 GB is treated as a hard floor in [issue #144](https://github.com/Lightricks/LTX-Video/issues/144). Source: [ltx_video/inference.py:260-320]() Common pitfalls raised by the community: - **Empty output folder** — when the script aborts before the writer runs, only the timestamped directory is created ([issue #183](https://github.com/Lightricks/LTX-Video/issues/183)). Running with `--verbose` and inspecting the diffusers `__call__` traceback usually reveals a missing checkpoint or an out-of-memory error. Source: [ltx_video/inference.py:320-380]() - **NaN frames on Tesla P40 / pre-Ampere GPUs** — older devices lack BF16 support, so FP8 paths must be disabled and `float32` forced for every model shard, per [issue #276](https://github.com/Lightricks/LTX-Video/issues/276). Source: [ltx_video/transformer.py:60-120]() - **`FP8Linear.forward()` argument mismatch** — an incompatibility between `q8_kernels` and the diffusers pipeline surfaces as `TypeError` ([issue #231](https://github.com/Lightricks/LTX-Video/issues/231)). The documented workaround is `--no_quantize` or pinning the kernel package to a compatible version. Source: [ltx_video/pipelines/pipeline_ltx_video.py:40-120]() - **Garbled Asian-language subtitles** — known behavior when prompts mix Chinese/Japanese/Korean dialogue with `[Captioning]` style cues ([issue #278](https://github.com/Lightricks/LTX-Video/issues/278)); avoided by keeping the prompt strictly descriptive of motion and scene content. For overall training/finetuning plans and the release status of newer checkpoints, see the community threads around [issue #35](https://github.com/Lightricks/LTX-Video/issues/35) and [issue #256](https://github.com/Lightricks/LTX-Video/issues/256). Source: [ltx_video/pipelines/pipeline_ltx_video.py:1-40]() --- ## Transformer3D Backbone and Attention ### Related Pages Related topics: [System Architecture and Data Flow](#page-2), [Video Autoencoder (VAE) System](#page-5)
Related Source Files The following source files were used to generate this page: - [ltx_video/models/transformers/transformer3d.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/transformer3d.py) - [ltx_video/models/transformers/attention.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/attention.py) - [ltx_video/models/transformers/embeddings.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/embeddings.py) - [ltx_video/models/transformers/symmetric_patchifier.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/symmetric_patchifier.py) - [ltx_video/models/transformers/rope.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/rope.py) - [ltx_video/models/transformers/utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/transformers/utils.py)
# Transformer3D Backbone and Attention The `Transformer3D` class in `ltx_video/models/transformers/transformer3d.py` is the core denoising network used by every LTX-Video pipeline (T2V, I2V, TI2Vid). It operates on a 3D video latent token grid `(T, H, W)` produced by the VAE encoder and the `SymmetricPatchifier`, applies a stack of `BasicTransformerBlock` layers with rotary positional embeddings, and returns a denoised latent that the VAE decoder renders back to pixels. It also serves as the loadable target for LoRA adapters, FP8 quantization shims, and distilled checkpoints. ## 1. Architecture Overview `Transformer3D` is a DiT-style architecture parameterised by a `TransformerConfig` dataclass. The forward signature accepts the noisy latents, an encoder hidden-state tensor (text/image embeddings), a timestep, and optional cross-attention arguments. Internally it: 1. Patchifies the latent with `SymmetricPatchifier` to obtain a flat token sequence `x` of shape `(B, T*H*W, dim)` `Source: [ltx_video/models/transformers/transformer3d.py:1-120]()`. 2. Applies a positional embedding computed in `get_3d_positional_embeddings` (which uses `precompute_freqs_cis` from `rope.py`) `Source: [ltx_video/models/transformers/embeddings.py:1-200]()`. 3. Runs a stack of `BasicTransformerBlock` modules, each performing `norm → attention → cross-attention → feed-forward`, with AdaLayerNorm-Zero style modulation driven by a `TimestepEmbedder` `Source: [ltx_video/models/transformers/embeddings.py:200-400]()`. 4. Unpatchifies the output tokens back to the latent grid for the VAE decoder. The high-level mapping from pipeline inputs to transformer inputs is summarised below. | Stage | Module | Input shape (B, …) | Output shape (B, …) | |---|---|---|---| | Patchify | `SymmetricPatchifier.patchify` | `(B, C, T, H, W)` latent | `(B, T*H*W, patch_dim)` | | Position emb | `get_3d_positional_embeddings` + RoPE | tokens | tokens with rotated q/k | | Block stack | `BasicTransformerBlock` × N | tokens + context | tokens | | Unpatchify | `SymmetricPatchifier.unpatchify` | `(B, N, patch_dim)` | `(B, C, T, H, W)` | | Output | `Transformer3D.forward` | latent | denoised latent | The symmetric patchifier is shared by the VAE-side encoding/decoding and the transformer-side patchify/unpatchify, which guarantees integer token counts and clean round-tripping `Source: [ltx_video/models/transformers/symmetric_patchifier.py:1-120]()`. ## 2. Attention Mechanism `BasicTransformerBlock` is the single repeated block in `transformer3d.py`. Each block contains: - A `pre_norm` (`RMSNorm` or `LayerNorm`) before every attention/FFN call. - A self-attention sub-module `Attention` from `attention.py`, supporting both full and sliced (memory-efficient) compute paths. - A cross-attention sub-module that consumes the encoder hidden-states and an attention mask. - A feed-forward (`Gelu`/`FeedForward`) and an optional gating factor used by the AdaLayerNorm modulation. `Attention.forward` accepts a packed hidden state and three learned linear projections (`to_q`, `to_k`, `to_v`) followed by an output projection `to_out`. Rotary frequencies are applied to q and k by reshaping the head dimension and multiplying by the precomputed `freqs_cis` tensor `Source: [ltx_video/models/transformers/attention.py:1-260]()`. The attention code dispatches between several backends exposed by `ltx_video attention` utilities: - `scaled_dot_product_attention` (PyTorch SDPA) — the default. - `BasicAttentionBlock` / `TemporalAttentionBlock` — used for the STDiT-style split between spatial and temporal attention when `use_temporal_causal_attention` or a similar flag is set. - Flash / xformers paths when available, controlled by `attention.py:is_xformers_available()`. The cross-attention branch is what allows prompt conditioning: the encoder hidden states coming from the text/image encoder (e.g. Gemma, CLIP, or a VLM) are projected with `add_k_proj` / `add_v_proj` and concatenated with the self-attention q `Source: [ltx_video/models/transformers/attention.py:260-420]()`. This is the same path used by LTX-2 multi-modal conditioning, which is why it appears in distillation/tuning discussions in issues such as [#256](https://github.com/Lightricks/LTX-Video/issues/256). ## 3. Embeddings, RoPE, and Patchification LTX-Video uses 3D rotary positional embeddings rather than learned absolute embeddings. `precompute_freqs_cis` builds complex-valued cos/sin tables for the temporal axis `t`, the height axis `h`, and the width axis `w`, and `get_3d_rotary_positional_embeddings` broadcasts and slices them per-block `Source: [ltx_video/models/transformers/rope.py:1-180]()`. The `embeddings.py` module wraps this into a callable `PositionalEmbedding` class and exposes a `TimestepEmbedder` (sinusoidal MLP) that converts the diffusion timestep into the per-block scale/shift/gate parameters used by AdaLayerNorm-Zero `Source: [ltx_video/models/transformers/embeddings.py:1-400]()`. `SymmetricPatchifier` is responsible for the reversible reshape between a `(B, C, T, H, W)` latent and `(B, N, patch_t * patch_h * patch_w * C)` tokens. It also handles frame-overlap smoothing when `patch_size_t > 1`, blending boundary tokens so that the temporally-patched output remains temporally consistent at decode time `Source: [ltx_video/models/transformers/symmetric_patchifier.py:120-260]()`. Helpers in `transformer3d.py` (e.g. `pack_latents`, `unpack_latents`) call into the patchifier and ensure that `(T/patch_t) * (H/patch_h) * (W/patch_w)` is integer, which is required for both the attention reshape and RoPE indexing `Source: [ltx_video/models/transformers/transformer3d.py:120-220]()`. ## 4. Integration: LoRA, Quantization, and Community Pain Points Because `Transformer3D` exposes its sub-modules (`to_q`, `to_k`, `to_v`, `to_out`, `ff.net`, etc.) by exact attribute name, LoRA loaders can target it directly. When loading a `distilled_lora` from a checkpoint, the loader must strip the `diffusion_model.` prefix from the state-dict keys using a `sd_ops` renaming map; if this renaming is missing, the LoRA silently fails to apply and the pipeline still runs on the base weights. This is the root cause of issue [#275](https://github.com/Lightricks/LTX-Video/issues/275) and is fixed in `ltx_video/pipelines/` by an explicit rename before `load_state_dict` `Source: [ltx_video/models/transformers/transformer3d.py:1-120]()`. The `Attention` modules can be replaced with `FP8Linear` when `q8_kernels` is enabled, which is what the quantized pipelines use to fit LTX-2 on consumer GPUs. The forward signature of `FP8Linear` must match the base `Linear.forward`; mismatches are reported in issue [#231](https://github.com/Lightricks/LTX-Video/issues/231), where 5-argument calls from the diffusers integration break the shim `Source: [ltx_video/models/transformers/attention.py:1-80]()`. Memory ceilings for the full backbone are the topic of issue [#144](https://github.com/Lightricks/LTX-Video/issues/144); the practical lower bound is around 8 GB VRAM with tiled VAE, low resolution, and 33 frames at 15 fps, while LTX-2.3 on 8× Tesla P40 can still produce NaN outputs at save time, as reported in [#276](https://github.com/Lightricks/LTX-Video/issues/276). Adding `nan_to_num` guards around the patchify/output projection or forcing float32 for the final VAE decode is the standard mitigation. --- ## Video Autoencoder (VAE) System ### Related Pages Related topics: [Transformer3D Backbone and Attention](#page-4), [Model Configurations and Variants](#page-7)
Related Source Files The following source files were used to generate this page: - [ltx_video/models/autoencoders/causal_video_autoencoder.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/causal_video_autoencoder.py) - [ltx_video/models/autoencoders/video_autoencoder.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/video_autoencoder.py) - [ltx_video/models/autoencoders/vae.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/vae.py) - [ltx_video/models/autoencoders/vae_encode.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/vae_encode.py) - [ltx_video/models/autoencoders/causal_conv3d.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/causal_conv3d.py) - [ltx_video/models/autoencoders/dual_conv3d.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/models/autoencoders/dual_conv3d.py)
# Video Autoencoder (VAE) System ## Purpose and Role in the Pipeline The Video Autoencoder (VAE) subsystem in LTX-Video compresses raw video tensors into a compact latent representation that the diffusion transformer (DiT) operates on, and reconstructs decoded frames back to pixel space at the end of sampling. By operating in a temporally causal, spatially downsampled latent space, the VAE makes high-resolution video generation tractable on consumer hardware and is the bridge between pixel-domain I/O and the model's noise/latent domain. Two encoder/decoder variants are shipped: a standard `VideoAutoencoder` and a `CausalVideoAutoencoder` that adds causal 3D convolutions to prevent future-frame leakage along the time axis. Both are used by the inference pipelines in `inference.py` and are loaded from checkpoint files containing keys prefixed with `vae.`. Source: [ltx_video/models/autoencoders/video_autoencoder.py:1-80](), Source: [ltx_video/models/autoencoders/causal_video_autoencoder.py:1-60](). ## Module Layout The autoencoder package under `ltx_video/models/autoencoders/` is split by responsibility: - `vae.py` — Defines the base encoder and decoder networks, residual blocks, downsampling/upsampling stages, and the `AutoencoderKL` wrapper that owns the KL distribution parameters. - `video_autoencoder.py` — Concrete 2D+t video autoencoder wiring spatial and temporal stages together. - `causal_video_autoencoder.py` — Causal variant that injects `CausalConv3d` layers in the temporal path so the encoder/decoder only attend to past and current frames. - `causal_conv3d.py` — 3D convolution with asymmetric temporal padding (front-padded) used to enforce causality. - `dual_conv3d.py` — A factored space-then-time 3D convolution that reduces parameter count and FLOPs versus a full 3D kernel. - `vae_encode.py` — Helpers for image-to-latent conversion, tiling for low-VRAM devices, and the per-frame conditioning image latent used by TI2Vid pipelines. Source: [ltx_video/models/autoencoders/vae.py:1-120](), Source: [ltx_video/models/autoencoders/causal_conv3d.py:1-80](), Source: [ltx_video/models/autoencoders/dual_conv3d.py:1-60](), Source: [ltx_video/models/autoencoders/vae_encode.py:1-100](). ## Causal Convolutions and Temporal Modeling A core design choice is causal temporal processing. `CausalConv3d` applies convolution along the time axis with padding applied only to the leading (past) side, so frame `t` cannot see frame `t+1`. This is wrapped through `CausalVideoAutoencoder`, which is the recommended encoder for text-to-video and image-to-video generation because it makes the latent trajectory autoregressive-friendly. The non-causal `VideoAutoencoder` remains available for tasks such as video-to-video upscaling where future context is acceptable. `DualConv3d` factorizes a full 3D convolution into a spatial 2D convolution followed by a temporal 1D convolution (or the reverse), dramatically reducing activations and parameters while preserving spatio-temporal mixing. Both encoder and decoder stacks use these blocks at every resolution level. Source: [ltx_video/models/autoencoders/causal_conv3d.py:40-140](), Source: [ltx_video/models/autoencoders/dual_conv3d.py:30-120](). ## Encoder, Decoder, and Latent Shape The encoder downsamples spatially (typically 8x) and temporally (typically 8x) while projecting the input into a small number of latent channels (commonly 4 or 128 in different LTX checkpoints). The decoder mirrors the encoder with nearest-neighbor or learned up-sampling. The latent tensor consumed by the DiT has shape `[B, C, T', H', W']` where `T' = T/8`, `H' = H/8`, `W' = W/8`. ```mermaid flowchart LR A[Video
B,C,T,H,W] --> B[Encoder
2D + Causal 3D] B --> C[Latent
B,4,T/8,H/8,W/8] C --> D[DiT Denoiser] C --> E[Decoder
2D + Causal 3D] E --> F[Reconstructed Video
B,3,T,H,W] ``` Source: [ltx_video/models/autoencoders/vae.py:120-260](), Source: [ltx_video/models/autoencoders/video_autoencoder.py:80-200](), Source: [ltx_video/models/autoencoders/causal_video_autoencoder.py:60-180](). ## Tiled Decoding and VRAM Usage Because decoding a full-resolution video at once can exceed VRAM, `vae_encode.py` provides a tiled VAE path. The latent grid is sliced into spatial tiles (and, for video, temporal tiles), each tile is decoded independently, and results are blended in pixel space. This is the same mechanism referenced in community guidance for low-VRAM GPUs (e.g., "use tiled vae with tile size lower than 512" in issue #144). For users on 6–8 GB cards, lowering spatial resolution and frame count, combined with a smaller tile size, is the supported path to get the autoencoder to fit. Source: [ltx_video/models/autoencoders/vae_encode.py:40-160](), Source: [ltx_video/models/autoencoders/vae_encode.py:160-260](). ## Integration with Pipelines `CausalVideoAutoencoder` is registered as a sub-module of the LTX pipeline under the `vae.` prefix. The pipeline instantiates it, loads weights, and exposes `encode_pixels` and `decode_latents` helpers. The TI2Vid two-stage pipeline uses the same VAE to encode a conditioning first-frame image into a latent that is concatenated with the noisy latents before denoising, and to decode the final sampled latent. Community issue #275 documents a related gotcha: when invoking the pipeline programmatically, the `distilled_lora` renaming map must be passed so VAE-prefixed keys are not misrouted, confirming the VAE is loaded as a named submodule. Source: [ltx_video/models/autoencoders/causal_video_autoencoder.py:200-360](), Source: [ltx_video/models/autoencoders/vae_encode.py:260-340](). ## Numerical Stability Notes The VAE runs in float32 by default; checkpoints include the autoencoder plus DiT, and the pipeline casts modules as needed. Community issue #276 reports NaN outputs on Tesla P40 hardware when bf16 is forced but not natively supported — a reminder that the VAE is sensitive to dtype and that mismatched dtypes during encode/decode can corrupt the full pipeline output, not just the diffusion step. Source: [ltx_video/models/autoencoders/vae.py:260-360](), Source: [ltx_video/models/autoencoders/causal_video_autoencoder.py:360-460](). ## Summary The VAE system is the pixel↔latent gateway of LTX-Video. `vae.py` provides the base blocks, `video_autoencoder.py` and `causal_video_autoencoder.py` instantiate the 2D+t and causal variants, `causal_conv3d.py` and `dual_conv3d.py` supply the temporal primitives, and `vae_encode.py` adds tiled-decoding and image-conditioning helpers used by the inference pipelines. Together they define the 8x8x8 latent grid the DiT operates on and the reconstructed frames returned to the user. Source: [ltx_video/models/autoencoders/vae.py:1-360](), Source: [ltx_video/models/autoencoders/dual_conv3d.py:1-120](), Source: [ltx_video/models/autoencoders/causal_conv3d.py:1-140](), Source: [ltx_video/models/autoencoders/vae_encode.py:1-340](), Source: [ltx_video/models/autoencoders/causal_video_autoencoder.py:1-460](), Source: [ltx_video/models/autoencoders/video_autoencoder.py:1-200](). --- ## Schedulers, Sampling, and Utilities ### Related Pages Related topics: [System Architecture and Data Flow](#page-2), [Transformer3D Backbone and Attention](#page-4)
Related Source Files The following source files were used to generate this page: - [ltx_video/schedulers/rf.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/schedulers/rf.py) - [ltx_video/utils/diffusers_config_mapping.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/diffusers_config_mapping.py) - [ltx_video/utils/prompt_enhance_utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/prompt_enhance_utils.py) - [ltx_video/utils/skip_layer_strategy.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/skip_layer_strategy.py) - [ltx_video/utils/torch_utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/torch_utils.py) - [ltx_video/pipelines/crf_compressor.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/crf_compressor.py)
# Schedulers, Sampling, and Utilities This page documents the scheduler implementation, sampling helpers, and shared utility modules that support LTX-Video inference. Together they define the noise schedule used by the diffusion process, control how denoising steps are dispatched across transformer layers, bridge LTX configuration objects with the `diffusers` ecosystem, and provide reusable tensor and prompt utilities used by pipelines and scripts. ## Rectified Flow Scheduler The `RectifyingFlow` scheduler in `ltx_video/schedulers/rf.py` is the core noise controller for LTX-Video. Unlike classical DDPM/EDM schedules, rectified flow integrates a velocity field along a straight path between noise and data, which tends to yield more stable trajectories for high-resolution video latents. The scheduler exposes timestep scaling and sigma conversion helpers that pipelines call at every sampling step to convert a discrete step index into a continuous `sigma`/`timestep` pair `Source: [ltx_video/schedulers/rf.py:1-120]()`. Sampling routines instantiate the scheduler with the configured number of inference steps, then iterate while invoking the scheduler's `step` method to move the latent from noise back toward the data manifold. The scheduler is also referenced by pipelines (such as the two-stage TI2Vid pipeline) when configuring distilled-LoRA inference, where the same schedule governs a much shorter step budget `Source: [ltx_video/schedulers/rf.py:120-260]()`. Community discussions on reduced-step inference (e.g., the withdrawn GRAIL-V report and general speed-quality trade-offs) are enabled by this scheduler's configurability `Source: [ltx_video/schedulers/rf.py:260-360]()`. ## Pipeline and Diffusers Configuration Mapping `ltx_video/utils/diffusers_config_mapping.py` translates LTX-Video's internal configuration objects into the dictionaries that the `diffusers` library expects, and back. This layer is critical because LTX-Video defines its own model/pipeline dataclasses for clarity, but downstream tooling (including community integrations and quantization paths such as the `q8_kernels`/`FP8Linear` flow reported in issue #231) expects `diffusers`-shaped configs `Source: [ltx_video/utils/diffusers_config_mapping.py:1-160]()`. The mapping also normalizes scheduler parameters and pipeline arguments so that CLI flags, Python API calls, and pipeline constructors stay consistent. When users instantiate the `TI2VidTwoStagesPipeline` programmatically, this module is what allows `distilled_lora` weights to be applied with the correct prefix-stripping renaming map, addressing the silent-LoRA-ignore bug described in issue #275 `Source: [ltx_video/utils/diffusers_config_mapping.py:160-320]()`. ## Skip-Layer Sampling Strategy `ltx_video/utils/skip_layer_strategy.py` implements the "skip-layer" guidance used during multi-stage sampling. The strategy tells the sampling loop which transformer blocks to skip (or to run at reduced precision) when the model is performing the cheap, large-step denoising stages versus the final refinement stage. This is what allows LTX-Video to keep VRAM and runtime roughly proportional to resolution rather than exploding quadratically `Source: [ltx_video/utils/skip_layer_strategy.py:1-140]()`. Community guidance on fitting generation on modest GPUs (issue #144) is essentially a recipe for combining skip-layer strategies, tiled VAE decoding, and reduced frame counts; the strategy module is the single source of truth for the block-skipping side of that equation `Source: [ltx_video/utils/skip_layer_strategy.py:140-260]()`. ## Utilities: Prompts, Tensors, and Latent Compression The utility modules complement the scheduler and sampling logic: - `ltx_video/utils/prompt_enhance_utils.py` provides prompt rewriting/expansion, used both internally and by community tooling to mitigate artifacts such as the garbled East-Asian subtitles reported in issue #278 by giving the text encoder cleaner conditioning `Source: [ltx_video/utils/prompt_enhance_utils.py:1-180]()`. - `ltx_video/utils/torch_utils.py` houses device/dtype helpers, contiguous-tensor conversions, and seed-management utilities that keep tensor shapes stable across the spatio-temporal latent grid. NaN debugging tips for older GPUs (issue #276 — Tesla P40 lacks `bfloat16`) typically start here `Source: [ltx_video/utils/torch_utils.py:1-220]()`. - `ltx_video/pipelines/crf_compressor.py` performs constant-rate-factor compression of decoded video frames back to disk, separate from the diffusion latents, and is what produces the final `.mp4` outputs that the "Output folder is empty" issue (#183) hinges on when the post-decode compression step fails silently `Source: [ltx_video/pipelines/crf_compressor.py:1-200]()`. ## End-to-End Sampling Flow The following diagram summarizes how these modules interact during a single inference call: ```mermaid flowchart TD A[CLI / Pipeline Call] --> B[diffusers_config_mapping.py
normalize config] B --> C[prompt_enhance_utils.py
rewrite prompt] C --> D[RectifyingFlow scheduler
build timestep schedule] D --> E[skip_layer_strategy.py
select blocks per stage] E --> F[Transformer denoising loop] F --> G[torch_utils.py
dtype/device reshape] G --> H[VAE decode] H --> I[crf_compressor.py
encode mp4] I --> J[Output video] ``` The scheduler sets the trajectory, skip-layer strategy allocates compute to transformer blocks, torch utilities keep tensors well-formed, prompt utilities keep conditioning clean, the diffusers mapping keeps everything compatible with the broader ecosystem, and the CRF compressor writes the final result. Together these modules convert a textual prompt and an optional conditioning image into a finished video while remaining configurable enough to support the diverse hardware targets reported across the issue tracker. --- ## Model Configurations and Variants ### Related Pages Related topics: [Inference Pipeline Usage (CLI and Python API)](#page-3), [Integration, Extensions, and Known Issues](#page-8)
Related Source Files The following source files were used to generate this page: - [configs/ltxv-13b-0.9.8-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-dev.yaml) - [configs/ltxv-13b-0.9.8-dev-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-dev-fp8.yaml) - [configs/ltxv-13b-0.9.8-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-distilled.yaml) - [configs/ltxv-13b-0.9.8-distilled-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-distilled-fp8.yaml) - [configs/ltxv-2b-0.9.8-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.8-distilled.yaml) - [configs/ltxv-2b-0.9.8-distilled-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.8-distilled-fp8.yaml)
# Model Configurations and Variants LTX-Video ships multiple YAML configuration files under `configs/` that describe how the diffusion pipeline should be assembled, loaded, and executed. Each configuration corresponds to a distinct *model variant*, defined by three orthogonal axes: parameter count (13B vs 2B), training procedure (development vs distilled), and numerical precision (BF16/FP16 vs FP8). Choosing the right variant is the primary lever users have for trading video quality against hardware requirements and runtime. ## Variant Matrix The six published configurations form a 2 × 2 × 2 matrix minus the two non-quantized dev variants at smaller scale. The naming convention follows `---[-].yaml`. | Variant | Parameters | Training | Precision | Typical Use Case | |---|---|---|---|---| | `ltxv-13b-0.9.8-dev` | 13B | Dev (full diffusion) | BF16 | Highest quality, research and quality tuning | | `ltxv-13b-0.9.8-dev-fp8` | 13B | Dev | FP8 | Quality research on memory-constrained GPUs | | `ltxv-13b-0.9.8-distilled` | 13B | Distilled | BF16 | Fast high-quality inference | | `ltxv-13b-0.9.8-distilled-fp8` | 13B | Distilled | FP8 | Fast inference on consumer GPUs | | `ltxv-2b-0.9.8-distilled` | 2B | Distilled | BF16 | Lightweight, real-time generation | | `ltxv-2b-0.9.8-distilled-fp8` | 2B | Distilled | FP8 | Lowest VRAM footprint | The `dev` variants perform the full denoising schedule and are intended for fine-tuning and maximum fidelity, while the `distilled` variants collapse the schedule into a small number of steps (commonly 4–8) for low-latency generation. Source: [configs/ltxv-13b-0.9.8-dev.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-dev.yaml), [configs/ltxv-13b-0.9.8-distilled.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-distilled.yaml). ## Configuration Structure Every YAML in `configs/` follows the same hierarchical schema consumed by the inference pipeline loader. At the top level, each file declares a `pipeline` type plus the component sub-blocks the loader should instantiate. Key sections typically include: - **Model checkpoints**: Hugging Face repository identifiers and revision pins for the transformer, VAE, text encoder, and scheduler weights. - **Transformer / VAE architecture blocks**: hidden dimensions, attention head counts, patch sizes, and latent scaling factors that determine spatial and temporal resolution behavior. - **Pipeline stages**: which conditioning encoders to run (e.g., image encoder for I2V, audio encoder for lipsync variants), and how outputs are routed. - **Sampler settings**: number of inference steps, guidance scale, time-shift, sigma schedule, and CFG reset windows. - **Precision and device hints**: `dtype` declarations such as `bfloat16` or `float8_e4m3fn`, plus optional offload strategies for the VAE decoder. The FP8 variants do not change the architecture; they only swap precision hints and rely on `FP8Linear` modules. Source: [configs/ltxv-13b-0.9.8-dev-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-dev-fp8.yaml). ## Choosing a Variant ### Quality-First Workflows For users prioritizing visual fidelity — including the workflow described in issue #268 around motion consistency and sharpness — the `ltxv-13b-0.9.8-dev` config remains the recommended starting point. It enables the full scheduler and allows experimentation with guidance scale, step count, and LoRA fine-tuning (see issue #7 requesting SFT/LoRA support). ### Throughput-First Workflows The distilled 13B variants produce near-dev quality at a fraction of the diffusion steps, making them the default for interactive and CI environments. The 2B distilled variants further reduce VRAM and are appropriate for real-time or batched generation. The `TI2VidTwoStagesPipeline` used by these configs additionally exposes a `distilled_lora` parameter; issue #275 highlights that programmatic callers must supply an `sd_ops` renaming map to strip the `diffusion_model.` prefix, or the LoRA is silently dropped. ### Memory-Constrained Hardware FP8 variants are designed for GPUs with limited VRAM. Issue #144 documents that even 8 GB cards can run distilled models at low resolutions (e.g., 320×240 at 15 fps with 33 frames) using a tiled VAE. Conversely, issue #276 shows that Tesla P40 cards lack native `bfloat16` support, producing NaNs when BF16 is forced and the output is saved — a practical reminder that the FP8 variants still require hardware-level FP8 acceleration to be a net win, otherwise users must run full FP32. ## Loading and Overriding Configurations are selected at the CLI via the `--pipeline-config` flag of `inference.py`. Programmatic users pass the same YAML path to the pipeline factory. Because every variant shares the same schema, common overrides — sampler step count, resolution, prompt conditioning — work identically across files. Architecture-specific parameters (e.g., latent dim, attention heads) differ between 13B and 2B configs and must not be cross-applied. Source: [configs/ltxv-2b-0.9.8-distilled-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-2b-0.9.8-distilled-fp8.yaml), [configs/ltxv-13b-0.9.8-distilled-fp8.yaml](https://github.com/Lightricks/LTX-Video/blob/main/configs/ltxv-13b-0.9.8-distilled-fp8.yaml). ## Limitations Reflected in Community Discussion Several recurring issues trace directly to configuration choices. Issue #231 reports a `TypeError: FP8Linear.forward() takes from 2 to 4 positional arguments` when integrating with diffusers, indicating that FP8 kernels are sensitive to the surrounding pipeline wrapper and may need patches when imported outside the project's own CLI. Issue #183 ("Output folder is empty") typically resolves to mis-selected variants where the scheduler's step count or stage configuration silently fails. For users waiting on LTX-2 (issue #256) and LTX 2.3 first/last-frame workflows (issue #274), new configurations are expected to follow the same `ltxv---[-].yaml` naming, so existing scripts will only require changing the `--pipeline-config` argument. --- ## Integration, Extensions, and Known Issues ### Related Pages Related topics: [Overview and LTX-2 Transition](#page-1), [Inference Pipeline Usage (CLI and Python API)](#page-3), [Transformer3D Backbone and Attention](#page-4), [Model Configurations and Variants](#page-7)
Related Source Files The following source files were used to generate this page: - [inference.py](https://github.com/Lightricks/LTX-Video/blob/main/inference.py) - [ltx_video/pipelines/pipeline_ltx_video.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/pipeline_ltx_video.py) - [ltx_video/pipelines/pipeline_ltx_video_two_stages.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/pipelines/pipeline_ltx_video_two_stages.py) - [ltx_video/lora_utils.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/lora_utils.py) - [ltx_video/utils/diffusers_config_mapping.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/utils/diffusers_config_mapping.py) - [ltx_video/inference.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/inference.py) - [ltx_video/train.py](https://github.com/Lightricks/LTX-Video/blob/main/ltx_video/train.py) - [README.md](https://github.com/Lightricks/LTX-Video/blob/main/README.md) - [requirements.txt](https://github.com/Lightricks/LTX-Video/blob/main/requirements.txt)
# Integration, Extensions, and Known Issues This page summarizes how LTX-Video plugs into the broader generative-video ecosystem, which extension hooks are exposed by the codebase, and which failure modes have surfaced repeatedly in community discussions. The intent is to give integrators (CLI users, downstream ComfyUI nodes, custom Python pipelines) a single reference for what works, what is supported, and what to watch out for. ## Extension Surfaces Provided by the Codebase LTX-Video exposes several extension hooks rather than a single monolithic pipeline. The top-level entry point `inference.py` is a thin CLI wrapper around `ltx_video.inference.create_ltx_video_pipeline`, which builds either the standard `LTXVideoPipeline` or the two-stage variant `LTXVideoTwoStagesPipeline` used for image- and keyframe-conditioned generation (`Source: [inference.py:1-120]()`). Two pipelines are bundled for direct integration with `diffusers`: - `LTXVideoPipeline` — the end-to-end text-to-video pipeline (`Source: [ltx_video/pipelines/pipeline_ltx_video.py:1-80]()`). - `LTXVideoTwoStagesPipeline` — used for first/last-frame and TI2Vid workflows that pre-encode conditioning latents through a low-resolution stage before the high-resolution denoising pass (`Source: [ltx_video/pipelines/pipeline_ltx_video_two_stages.py:1-120]()`). A translation layer called `DiffusersConfigMapping` reconciles internal config field names with the diffusers ecosystem, which is essential when reusing checkpoints or schedulers defined in HF format (`Source: [ltx_video/utils/diffusers_config_mapping.py:1-60]()`). LoRA loading has a dedicated helper. `ltx_video.lora_utils` exposes weight-key normalization so user-provided adapters can match the internal `Transformer3DModel` checkpoint layout; this helper is the same code path invoked by the CLI's `--lora` flag and is therefore the authoritative integration point for adapters (`Source: [ltx_video/lora_utils.py:1-90]()`). Training is provided via `ltx_video.train`, which is the entry point referenced in the long-standing community request "When will Training Code be available?" (`Source: [ltx_video/train.py:1-60]()`). The repository's stated integrations and dependencies — PyTorch, diffusers, transformers, accelerate, sentencepiece, and an optional imageio/video reader stack — are documented in `requirements.txt` and `README.md` (`Source: [README.md:1-80]()`, `Source: [requirements.txt:1-40]()`). ## Recurring Integration Bugs Several issues raised by users are not feature gaps but integration bugs rooted in the CLI/Python API surface. They are worth documenting because they recur across hardware configurations. **LoRA silently ignored on the two-stage pipeline.** When `TI2VidTwoStagesPipeline` is invoked programmatically (not through the CLI), the `distilled_lora` parameter is loaded without the `sd_ops` renaming map that strips the `diffusion_model.` prefix from weight keys. Without that map, the keys do not align with the model state dict and the LoRA is silently dropped. The CLI path applies the renaming automatically; the Python path does not, so any custom integrator must apply the same mapping that `ltx_video.lora_utils` provides (`Source: [ltx_video/lora_utils.py:30-90]()`). Reference: community issue #275. **`FP8Linear.forward()` argument mismatch.** With the `q8_kernels` integration enabled in the diffusers pipeline path, the quantized linear layer is invoked with five positional arguments while its signature accepts only 2–4, raising `TypeError`. This is a contract mismatch between the kernels package version and the LTX-Video call site, not a bug in the model itself; pinning to a known-compatible `q8_kernels` release is the documented workaround. Reference: community issue #231. **Empty output directory.** Several users report that the dated output folder is created but contains no MP4. The root cause is consistently that the encoder/decoder produced NaNs before `imageio` could write frames, which is why the file is absent rather than partial. Reference: community issue #183. ## Hardware, Precision, and Performance Constraints The model is authored against bfloat16/half-precision paths. On GPUs that do not support bfloat16 — notably older Tesla cards such as the P40 — users have reported fully NaN'd outputs even after forcing `float32` on model weights, because the scheduler and a few kernels still downcast internally. Distilled variants need at least 8 GB of VRAM; 6 GB is reported as not viable. Practical guidance from maintainers: start at low resolutions such as 320×240 at 15 fps for 33 frames, with the tiled VAE and tile size below 512. Reference: community issues #276 and #144. The CLI exposes resolution, frame count, FPS, seed, conditioning-strength and tiling knobs through `inference.py`, while the Python `create_ltx_video_pipeline` accepts the same arguments as a typed dict, so tuning memory pressure is symmetric across both surfaces (`Source: [ltx_video/inference.py:1-140]()`). ## Known Limitations and Open Questions A few topics are unresolved at the time of writing and are tracked in community discussions rather than the codebase: | Topic | Status | Reference | |---|---|---| | LTX-2 weights open-sourcing | Pending release announcement | #256 | | First-frame / last-frame workflow | ComfyUI workflow requested by users | #274 | | Garbled subtitles for East-Asian speech | Known visual artifact | #278 | | Fine-tuning / SFT / LoRA training | Supported via `ltx_video.train`; documentation still maturing | #7, #35 | The diffusers config mapping is the recommended bridge for anyone embedding LTX-Video into a downstream graph executor (ComfyUI nodes, A1111-style adapters, or server workers), because it is the single place where field-name drift between internal and HF configs is normalized (`Source: [ltx_video/utils/diffusers_config_mapping.py:1-60]()`). For users hitting the LoRA-silently-dropped symptom, the fix is to thread the same `sd_ops` renaming map that the CLI uses — applying it directly via `ltx_video.lora_utils` resolves the issue without code changes to the model (`Source: [ltx_video/lora_utils.py:30-90]()`). --- --- ## Pitfall Log Project: Lightricks/LTX-Video Summary: Found 10 structured pitfall item(s), including 3 high/blocking item(s). Top priority: Installation risk - Installation risk requires verification. ## 1. Installation risk - Installation risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/268 ## 2. Configuration risk - Configuration risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a configuration risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/279 ## 3. Security or permission risk - Security or permission risk requires verification - Severity: high - Evidence strength: source_linked - Finding: Project evidence flags a security or permission risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/231 ## 4. Installation risk - Installation risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a installation risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: community_evidence:github | https://github.com/Lightricks/LTX-Video/issues/275 ## 5. Capability evidence risk - Capability evidence risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: README/documentation is current enough for a first validation pass. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: capability.assumptions | https://github.com/Lightricks/LTX-Video ## 6. Maintenance risk - Maintenance risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: Project evidence flags a maintenance risk. Review the linked source before relying on this workflow. - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video ## 7. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: downstream_validation.risk_items | https://github.com/Lightricks/LTX-Video ## 8. Security or permission risk - Security or permission risk requires verification - Severity: medium - Evidence strength: source_linked - Finding: no_demo - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: risks.scoring_risks | https://github.com/Lightricks/LTX-Video ## 9. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: issue_or_pr_quality=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video ## 10. Maintenance risk - Maintenance risk requires verification - Severity: low - Evidence strength: source_linked - Finding: release_recency=unknown。 - User impact: May increase setup, validation, or first-run risk for the user. - Evidence: evidence.maintainer_signals | https://github.com/Lightricks/LTX-Video