DOCSEARCHbyAlgolia
Submit Docs

FastVideo Documentation

repository·main·Indexed 26 days ago

https://github.com/hao-ai-lab/fastvideo

FastVideo is a unified post-training and real-time inference framework for accelerated video generation, supporting bidirectional and autoregressive models. Version 0.2.0 includes the Dreamverse application, featuring a backend for video segment generation and a Next.js frontend. The framework supports CUDA 12.6 and 13.0, provides Docker deployment options, and includes a PromptEnhancer for guided continuation and autonomous expansion of video clips.

Tokens
88.3K
Snippets
238
Records
527
Agent score
87%

What's inside FastVideo

  1. Overview of FastVideo

    main
    FastVideo is a unified inference and post-training framework designed for accelerated video generation using diffusion models. It provides an end-to-end pipeline covering data preprocessing, model training, fine-tuning, distillation, and inference. The framework is modular and extensible, supporting both training-free and post-training optimizations.

  2. Overview of the FastVideo Modular Training Framework

    main
    The fastvideo/train/ directory contains the new modular training infrastructure for FastVideo. Unlike the legacy fastvideo/training/ system which used monolithic per-model pipeline scripts, the new framework is composable. It allows developers to combine model plugins with specific training method classes using hierarchical YAML configurations.

  3. Overview of FastVideo features

    main

    FastVideo is a unified post-training and real-time inference framework for accelerated video generation. Key capabilities include:

    • Post-training Support: Full and LoRA finetuning for video DiTs, data preprocessing, DMD2 stepwise distillation, Sparse attention (VSA), and Causal distillation via Self-Forcing.
    • Inference Optimizations: Sequence Parallelism for distributed inference and multiple attention backends.
    • Hardware Support: Optimized for H100, A100, and 4090 across Linux, Windows, and MacOS.
    • Realtime Generation: Includes Dreamverse, a platform for realtime video generation and "vibe directing" that can be deployed locally, via Docker, or on serverless Modal.

  4. Understand FastVideo Server Integration Contracts

    main

    FastVideo provides a typed public API (fastvideo.api) designed to be consumed by different server-class integrations. These integrations share a single execution substrate to maintain consistency. The three primary integration patterns are:

    1. Stateless OpenAI: Uses HTTP POST /v1/videos. It maps VideoGenerationsRequest to GenerationRequest merged onto ServeConfig.default_request. It is stateless, with an optional ContinuationState round-trip.
    2. Streaming WebSocket: Uses WebSocket JSON and binary fMP4. It processes a GenerationRequest per segment and uses a server-held SessionStore with snapshot-on-demand.
    3. Dynamo native backend: Uses Dynamo RPC. It maps NvCreateVideoRequest via an adapter to GenerationRequest. State is currently aggregated, with plans for disaggregation via ContinuationState.

    All integrations consume the same underlying fastvideo.api surface.

  5. Data requirements for FastVideo finetuning

    main

    FastVideo uses precomputed embeddings and latents to reduce GPU memory usage by eliminating the need to load the text encoder and VAE during training. The required components depend on the model type:

    Text-to-Video (T2V) Finetuning

    • Text embeddings: Precomputed embeddings (e.g., T5 or LLaMA) stored as numpy arrays in Parquet files.
    • Video latents: VAE-encoded representations of the training videos.

    Image-to-Video (I2V) Finetuning

    In addition to the T2V requirements, certain architectures (like Wan2.1 and Wan2.2 A14B) require:

    • First frame latent: VAE-encoded representation of the first frame used for conditioning.
    • CLIP features: Image embeddings from a CLIP vision encoder for the conditioning frame.

  6. Understand Dreamverse Architecture and Runtime

    main

    Dreamverse consists of two primary components:

    1. Frontend: A Next.js application located at apps/dreamverse/web/ that manages UI state, websocket connections, and user interaction flows.
    2. Server: A Python FastAPI runtime located at apps/dreamverse/dreamverse/ that manages generation state, prompt rewriting, safety, GPU execution, and websocket session semantics.

    Communication occurs via HTTP and a single websocket on the /ws endpoint. The server is the authoritative source of truth for generation state and prompt memory, while the frontend manages local UI state and proposes prompt changes.

  7. Use the Core FastVideo API for media generation

    main

    FastVideo provides a high-level interface for loading media generation pipelines and controlling the generation process through three primary components:

    • fastvideo.VideoGenerator: The main entry point used to load a pipeline and perform high-level media generation.
    • fastvideo.PipelineConfig: A configuration object used to describe the specific components (models, schedulers, etc.) required by a pipeline.
    • fastvideo.SamplingParam: A configuration object used to define sampling parameters at generation time (e.g., guidance scale, steps).

  8. Supported FastVideo Attention Kernels

    main

    FastVideo provides several optimized attention kernels for video generation:

    • Video Sparse Attention (VSA): A sparse attention mechanism that selects top-k blocks.
    • Sliding Tile Attention (STA): Kernel support is located in the fastvideo-kernel package. Note that the full FastVideo STA pipeline workflow is archived in the sta_do_not_delete directory.
    • Attn-QAT Training: Provides Runtime-JIT Triton forward and backward kernels designed for role-local quantization-aware training.

  9. Understand the Dreamverse Architecture and Responsibility Split

    main

    Dreamverse is designed as a local-first application where the browser communicates with a local control plane rather than directly with cloud providers. The architecture is split into three primary layers:

    1. Frontend (apps/dreamverse/web)

    • Responsibilities: UI state, local interaction state, selecting compute modes, displaying cost/health/status, and local forms for provider configuration. It sends prompt and rewrite requests to the local controller.
    • Constraints: Must not own provider credentials, provider API calls, or runtime lifecycle.

    2. Controller (apps/dreamverse/controller)

    • Responsibilities: Local-only storage of provider credentials, compute mode selection, provisioning/monitoring runtimes, and reverse proxying HTTP/Websocket traffic from the frontend to the active runtime.
    • Constraints: Must not own prompt rewrite logic, seed prompt memory, or generation queue behavior.

    3. Runtime (apps/dreamverse/dreamverse or runtime/)

    • Responsibilities: Authoritative owner of /ws session state, prompt rewrite execution, prompt safety, seed prompt memory, and generation orchestration.
    • Constraints: Must remain provider-agnostic and follow a stable contract.

  10. Understand the FastVideo training file structure

    main

    The training infrastructure is organized into several key directories within fastvideo/train/:

    • entrypoint/: Contains train.py, the CLI entrypoint used with torchrun.
    • trainer.py: The main orchestrator for the training loop.
    • models/: Contains model definitions (e.g., wan/ for Wan 2.1 T2V and Wan causal models).
    • methods/: Contains training algorithms such as distribution_matching/ (DMD2, Self-Forcing) and fine_tuning/ (SFT, DFSFT).
    • callbacks/: Contains pluggable hooks like grad_clip.py, ema.py, and validation.py.
    • utils/: Contains core utilities for configuration parsing, model/method building, optimization, checkpointing, and data loading.

  11. Understand the training directory layout

    main

    The examples/train/ directory is organized as follows:

    • configs/: Contains single-step training configurations categorized by method and model.
    • scenario/: Contains multi-step end-to-end training pipelines.
    • run.sh: The launcher script for single-node training.
    • run_slurm.sh: The launcher script for multi-node Slurm training.