LTX-Video

LTX-Video is a DiT-based video generation model supporting high-fidelity video production. Key capabilities include:

• Generation Modes: Text-to-video, image-to-video, multi-keyframe conditioning, keyframe-based animation, video extension (forward and backward), and video-to-video transformations.
• Performance: Can generate up to 50 FPS videos at native 4K resolution.
• Audio: Supports synchronized audio and video generation (notably in the LTX-2 model).
• Integration: Compatible with ComfyUI and Diffusers.

Specialized control models for Pose, Depth, and Canny control are available via the ComfyUI-LTXVideo repository. These models enable precise control over the generation process.

An example ComfyUI workflow for these control types can be found here: ic-lora.json.

For users with NVIDIA ADA GPUs (e.g., RTX 40-series), LTX-VideoQ8 offers an 8-bit optimized version of the model designed for speed.

• Performance: Up to 3X speed-up with no accuracy loss.
• VRAM Efficiency: Can generate 720x480x121 videos in under a minute on an RTX 4060 (8GB VRAM).
• Integration: A diffusers integration is available via sayakpaul/q8-ltx-video.

Detailed implementation can be found in the LTX-VideoQ8 repository.

To run LTX-Video on your local machine, clone the repository and install the package with the [inference] extra.

Requirements:

• Python 3.10.5
• CUDA 12.2 (tested)
• PyTorch >= 2.1.2
• For macOS: MPS tested with PyTorch 2.3.0 (supports PyTorch == 2.3 or >= 2.6)

Optional Performance Boost: For Ada architecture and later, you can install FP8 kernels to improve performance.

git clone https://github.com/Lightricks/LTX-Video.git
cd LTX-Video

# create env
python -m venv env
source env/bin/activate
python -m pip install -e .[inference]

LTX-2 is the next-generation foundation model from Lightricks, designed for synchronized audio and video generation. It is now the primary focus of LTX development.

Key LTX-2 features:

• Synchronized Audio+Video: Generates motion, dialogue, ambience, and music in one coherent process.
• Advanced Control: Supports multiple keyframes, IC-LoRA control models, and standard LoRA for style customization.
• Multiscale Pipelines: Includes a latent upsampler.
• Integration: Built into ComfyUI core.

For the latest development and documentation, visit the LTX-2 repository or docs.ltx.video.

The LTX-Video-Trainer repository allows for fine-tuning both the 2B and 13B model variants. It supports:

• Full Fine-tuning: Complete model updates.
• LoRA (Low-Rank Adaptation): Efficient training for specific use cases, including:
  • Control LoRAs: Training custom control models for depth, pose, and Canny.
  • Effect LoRAs: Creating specialized visual effects and transformations.

Detailed training instructions are located in the LTX-Video-Trainer README.

TeaCache is a training-free caching approach that accelerates LTX-Video inference by up to 2x without significant visual quality degradation. It works by leveraging timestep differences across model outputs.

• No Retraining: Works directly with existing LTX-Video models.
• Configurable: Offers adjustable trade-offs between inference speed and visual quality.

Implementation details are available in the TeaCache4LTX-Video repository.

The community-driven ComfyUI-LTXTricks project provides additional nodes for advanced control over LTX Video models within ComfyUI. It supports several specialized techniques:

• RF-Inversion: Implements RF-Inversion for video manipulation.
• RF-Edit: Implements RF-Solver-Edit.
• FlowEdit: Implements FlowEdit for video editing.
• I+V2V: Enables Video-to-Video generation using a reference image.
• Enhance: Provides a partial implementation of Spatiotemporal Skip Guidance (STG).
• Interpolation and Frame Setting: Allows precise control of latents per frame.

Refer to the ComfyUI-LTXTricks repository for specific example JSON workflows for each feature.

For optimal results, write prompts as a single flowing paragraph (under 200 words) using a chronological, cinematic description style.

Recommended Structure:

1. Start with the main action in one sentence.
2. Add specific details about movements and gestures.
3. Describe character/object appearances precisely.
4. Include background and environmental details.
5. Specify camera angles and movements.
6. Describe lighting and colors.
7. Note any sudden events.

Automatic Enhancement: If using LTXVideoPipeline directly, set enhance_prompt=True to enable automatic prompt enhancement.

Import infer and InferenceConfig from ltx_video.inference to integrate the model directly into your Python applications.

from ltx_video.inference import infer, InferenceConfig

infer(
    InferenceConfig(
        pipeline_config="configs/ltxv-13b-0.9.8-distilled.yaml",
        prompt=PROMPT,
        height=HEIGHT,
        width=WIDTH,
        num_frames=NUM_FRAMES,
        output_path="output.mp4",
    )
)

The InferenceConfig dataclass defines the parameters for video generation. Key fields include:

• prompt: The text prompt for generation.
• height / width: Output resolution.
• num_frames: Number of frames to generate.
• frame_rate: FPS for the output video.
• seed: Random seed for reproducibility.
• negative_prompt: Prompt for undesired features.
• conditioning_media_paths: List of paths to conditioning images or videos.
• conditioning_strengths: List of strengths (0-1) for each conditioning item.
• conditioning_start_frames: List of frame indices where each conditioning item starts.

If you use LTX-Video in your research, please cite the following paper:

@article{HaCohen2024LTXVideo,
  title={LTX-Video: Realtime Video Latent Diffusion},
  author={HaCohen, Yoav and Chiprut, Nisan and Brazowski, Benny and Shalem, Daniel and Moshe, Dudu and Richardson, Eitan and Levin, Eran and Shiran, Guy and Zabari, Nir and Gordon, Ori and Panet, Poriya and Weissbuch, Sapir and Kulikov, Victor and Bitterman, Yaki and Melumian, Zeev and Bibi, Ofir},
  journal={arXiv preprint arXiv:2501.00103},
  year={2024}
}