DOCSEARCHbyAlgolia
Submit Docs

LTX-2 Documentation

repository·main·Indexed 27 days ago

https://github.com/lightricks/ltx-2

A DiT-based (Diffusion Transformer) audio-video foundation model for high-fidelity video generation with synchronized audio. The repository includes ltx-core (v1.1.7), the core implementation featuring an asymmetric dual-stream architecture with 14B video and 5B audio parameters, and ltx-kernels, which provides custom CUDA/C++ extensions for blockwise FP8 GEMM, All2All communication, and fused element operations. It supports FP8 quantization, LoRA adapters via SingleGPUModelBuilder, and memory-constrained inference through StreamingModelBuilder.

Tokens
45.8K
Snippets
104
Records
237
Agent score
94%

What's inside LTX-2

  1. Overview of LTX-2 Trainer

    main

    The ltx-trainer package provides tools and scripts for training and fine-tuning the LTX-2 audio-video generation model. It supports various training methods and conditioning frameworks, including:

    • Training Methods: LoRA training and full fine-tuning.
    • Conditioning Frameworks:
      • Text-to-video
      • Text-to-audio
      • Image-to-video
      • Video extension
      • Audio extension
      • Video inpainting
      • Audio inpainting
      • Video outpainting
      • IC-LoRA (for video, audio, and joint audio-video references)
      • Audio-to-video
      • Video-to-audio

  2. Overview of LtxTrainerConfig sub-configurations

    main

    The LtxTrainerConfig class is the primary configuration object used by the trainer. It is composed of several specialized sub-configuration models to manage different aspects of the training process:

    • ModelConfig: Base model and training mode settings
    • LoraConfig: LoRA training parameters
    • TrainingStrategyConfig: Training strategy settings (flexible conditioning framework)
    • OptimizationConfig: Learning rate, batch sizes, and scheduler settings
    • AccelerationConfig: Mixed precision and quantization settings
    • DataConfig: Data loading parameters
    • ValidationConfig: Validation and inference settings
    • CheckpointsConfig: Checkpoint saving frequency and retention settings
    • HubConfig: Hugging Face Hub integration settings
    • WandbConfig: Weights & Biases logging settings
    • FlowMatchingConfig: Timestep sampling parameters

  3. Overview of LTX-2 Pipeline Types

    main

    The ltx-pipelines package provides high-level implementations for generating audio-video content. Supported pipeline tasks include:

    • Text-to-video
    • Image-to-video
    • Video-to-video
    • Audio-to-video
    • Keyframe interpolation
    • Retake

    These pipelines are built using components from ltx-core (such as schedulers, guiders, noisers, and patchifiers) and support features like FP8 transformers, LoRA integration, and two-stage generation for high-quality output.

  4. Understand the LTX-2 Asymmetric Dual-Stream Architecture

    main
    LTX-2 is an asymmetric dual-stream diffusion transformer that jointly models video and audio signals. It uses a 14B-parameter video stream (3D RoPE for spatiotemporal dynamics) and a 5B-parameter audio stream (1D RoPE for temporal dynamics). The two streams share 48 transformer blocks and interact via bidirectional cross-modal attention and Cross-Modality AdaLN to ensure synchronization (e.g., lip-sync and environmental acoustics).

  5. Understand Multi-GPU (MGPU) Inference constraints

    main

    LTX-2 Multi-GPU (MGPU) inference is designed as a latency reduction tool, not a memory expansion tool.

    Key Constraints:

    • Memory: MGPU cannot be used to fit models that exceed a single GPU's capacity. The mutable working copy of the transformer is a full replica on every GPU. To fit larger models, use FP8 quantization or weight offloading instead.
    • Topology: Supports single machine only. It requires one process per GPU, with MASTER_ADDR=localhost. Multi-node setups are not supported.
    • Weight Layout: While the working copy is replicated, the immutable clean (pre-LoRA) weights are sharded across GPUs (ShardedSD, approximately 1/world_size per rank).
    • Parallelism: Sequence parallelism splits activation memory across ranks.

  6. Understand the LTX-2 Flexible Training Strategy

    main

    The LTX-2 trainer uses a unified flexible training strategy (name: "flexible") to support all training modes via configuration. Instead of separate strategy classes, modes are defined by setting is_generated on each modality and adding optional conditions.

    • is_generated: true: The modality is denoised during training and contributes to the loss (the model learns to generate this).
    • is_generated: false: The modality is frozen (sigma=0, no noise, no loss) and acts as cross-modal conditioning.
    • Requirement: At least one modality must have is_generated: true.
    • Audio Constraints: Audio does not support first_frame or spatial_crop conditions. Supported audio conditions are prefix, suffix, mask, and reference.

  7. LTX-2 Package Overview

    main

    The LTX-2 repository is a monorepo containing three primary packages:

    • ltx-core: Contains the core model implementation, inference stack, and utilities.
    • ltx-pipelines: Provides high-level pipeline implementations for text-to-video, image-to-video, and other generation modes.
    • ltx-trainer: Provides tools for training and fine-tuning, including LoRA, full fine-tuning, and IC-LoRA.

  8. Understand Tiled Data Parallelism (TDP)

    main

    Tiled Data Parallelism (TDP) is an approximation technique used for upscaling video to resolutions higher than what the model was trained on. It splits the latent (frames, height, width) into tiles and distributes them across GPUs. Each rank runs the full transformer on its assigned tile(s), and overlapping regions are blended using trapezoidal masks.

    Key Characteristics:

    • Not bit-faithful: Unlike sequence parallelism, TDP is an approximation because each tile is denoised with only local context.
    • Upscaling focus: TDP should be used as a second stage (upscaler) rather than a first stage. Starting from pure noise with TDP causes tiles to produce unrelated content.
    • Performance: It can be faster than single-GPU execution because it reduces the quadratic self-attention cost per tile.
    • Audio Warning: Do not use the TDP stage's audio output. Audio is processed untiled on every tile and then averaged, which does not result in a meaningful audio latent. Always use the audio from the first (Sequence Parallel) stage and keep it frozen through the TDP upscale.

  9. Use RetakePipeline to regenerate specific video regions

    main

    The RetakePipeline allows you to regenerate a specific time region of an existing video while keeping the rest unchanged. It uses a temporal region mask to denoise only the specified [start_time, end_time] window.

    Required CLI arguments:

    • --video-path
    • --start-time
    • --end-time

    Constraints:

    • Source video frame count must satisfy the 8k+1 format (e.g., 97, 193).
    • Resolution must be multiples of 32.

  10. Use TI2VidOneStagePipeline for prototyping

    main

    The TI2VidOneStagePipeline is a single-stage generation pipeline (no upsampling) with multimodal guidance and image conditioning support. It is faster but produces lower resolution output (typically 512x768).

    Note: This pipeline is primarily for educational purposes and quick prototyping; use two-stage pipelines for production quality.

  11. Optimize memory footprint with FP8 Quantization

    main

    To reduce GPU memory usage, use FP8 quantization. You must set the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True when launching your process.

    Available Quantization Policies

    PolicyCLI FlagDescription
    FP8 Cast--quantization fp8-castDowncasts transformer linear weights to FP8 during loading; upcasts on the fly during inference. No extra dependencies.
    FP8 Scaled MM--quantization fp8-scaled-mmUses FP8 scaled matrix multiplication via PyTorch's torch._scaled_mm. Best performance on Hopper+ GPUs with native FP8 support.

    Usage via CLI

    # FP8 Cast
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
    --quantization fp8-cast --checkpoint-path=...

# FP8 Scaled MM
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
    --quantization fp8-scaled-mm --checkpoint-path=...

    Usage Programmatically

    Pass a QuantizationPolicy to your pipeline class using the appropriate build function from ltx_core.

    from ltx_core.quantization.fp8_cast import build_policy as build_fp8_cast_policy
# Alternative:
# from ltx_core.quantization.fp8_scaled_mm import build_policy as build_fp8_scaled_mm_policy

pipeline = TI2VidTwoStagesPipeline(
    checkpoint_path=ltx_model_path,
    distilled_lora=distilled_lora,
    spatial_upsampler_path=upsampler_path,
    gemma_root=gemma_root_path,
    loras=[],
    quantization=build_fp8_cast_policy(ltx_model_path),
)
pipeline(...)