DOCSEARCHbyAlgolia
Submit Docs

WanGP Documentation

repository·main·Indexed 27 days ago

https://github.com/deepbeepmeep/wan2gp

An open-source super app providing a unified interface for generative models (video, image, audio, and TTS), optimized for lower-end hardware. Includes detailed guides on using FACodec for speech encoding and quantization, zero-shot voice conversion via FACodec V2 and FACodecRedecoder, and the implementation of Seed-VC for voice, singing, and accent conversion including training and fine-tuning workflows.

Tokens
64.2K
Snippets
100
Records
315
Agent score
93%

What's inside WanGP

  1. Overview of Seed-VC capabilities

    main

    Seed-VC is a voice conversion model supporting zero-shot voice conversion, zero-shot real-time voice conversion, and zero-shot singing voice conversion. It allows for voice cloning using only 1 to 30 seconds of reference audio without requiring training.

    Key features include:

    • Zero-shot capabilities: Voice cloning from short reference clips.
    • Real-time conversion: Suitable for online meetings, gaming, and live streaming, with an algorithmic latency of approximately 300ms and device-side latency of approximately 100ms.
    • Fast Fine-tuning: Supports additional fine-tuning with minimal data (as little as 1 utterance per speaker) and high speed (e.g., 100 steps in ~2 minutes on a T4 GPU).

  2. Overview of WanGP capabilities

    main

    WanGP is an open-source super app designed to provide access to various generative models across multiple modalities, specifically optimized for users with limited hardware (the 'GPU Poor').

    Supported Modalities and Models

    • Video: Wan 2.1/2.2 (and derivatives), LTX-2, Hunyuan Video 1/1.5, LongCat, Kandinsky, LTXV, MagiHuman.
    • Image: Qwen Image, Z-Image, Flux 1/2 (Klein, Chroma), HiDream.
    • Audio / TTS: Qwen3 TTS, Ace Step 1/2/XL, Omnivoice, Index TTS2, KugelAudio, HearMula, Chatterbox.

    Key Hardware Features

    • Low VRAM: Supports running select models with as little as 6 GB of VRAM.
    • Nvidia Support: Compatible with RTX 10XX, 20XX, and newer cards.
    • AMD Support: Supports RDNA 4, 3, 3.5, and 2 hardware.
    • Quantization: Supports int8, fp8, gguf, NV FP4, and Nunchaku formats.
    • Automatic Downloads: Architecture-aware downloads fetch model files optimized for your specific hardware.

  3. Overview of WanGP Settings Schema

    main

    WanGP generation settings are JSON-serializable values used by wgp.py and the Python API in shared/api.py.

    Schema Hierarchy:

    1. The baseline schema is defined in models/_settings.json.
    2. Model defaults in defaults/*.json and finetunes/*.json override the baseline.
    3. Handler code may further update or hide settings based on the selected model definition.

    Best Practice: Use an exported settings file from a specific model as a template for accuracy.

  4. Understand the Core Editor architecture

    main

    The Core Editor is built around several specialized classes:

    • ImageEditor: The main entry point for initialization, tool management, and rendering.
    • CommandManager: Manages the undo/redo stacks using the Command pattern.
    • LayerManager: Handles layer textures, z-index, and the active layer.
    • EditorState: A reactive state container (using Svelte stores/springs) that maintains scale, position, and tool information, notifying subscribers of changes.
    • Tool Interface: The contract for all interactive tools.

  5. WanGP built-in creation and post-processing tools

    main

    WanGP includes several tools for managing and enhancing generative outputs:

    Management & Workflow

    • Galleries: Browse and reuse video, image, and audio generations.
    • Reusable Settings: Extract settings from generations to create and share templates.
    • Generation Queue: Line up multiple jobs for later processing.
    • Headless Mode: Launch batches of images, videos, or audio via the command line.
    • WanGP API: Integrate generative capabilities into external applications.

    Enhancement & Post-processing

    • Prompt Enhancer: Model-specific syntax improvement.
    • Input Preparation: Mask editor, background remover, pose/depth/flow extractors, speaker diarization, and background noise/song remover.
    • Upsampling: Temporal and spatial upsampling using RIFE, FlashVSR, and Lanczos.
    • Audio Post-processing: Soundtrack generation with MMAudio, voice replacement with SeedVC, and video remuxing.
    • Deepy Offline Agent: Orchestrates background tasks like transcription, video splitting, and color-frame generation.

  6. Understand the Image Editor architecture and data flow

    main

    The editor is built with PIXI.js (rendering) and Svelte (UI/State).

    Data Flow

    1. User Interaction: Occurs via UI or canvas.
    2. Tool Handling: The active tool processes the interaction.
    3. Command Creation: An operation that modifies state creates a Command.
    4. Command Execution: The command is executed and registered with the Command Manager.
    5. State Update: The editor state is updated via Svelte stores.
    6. Rendering: The PIXI.js pipeline re-renders the layers.

    Layer Structure

    • Background Layer: The base image.
    • Drawing Layers: User modifications and drawings.
    • UI Layer: Overlaying UI elements.

  7. Implement an Audio Processor Plugin

    main

    To create a custom audio processor, implement a handler class that follows the Audio Processor Plugin API. The handler must define its capabilities via query_audio_processor_def() and implement the specific processing methods required for its type (soundtrack, voice_replacement, or audio_edit).

    Processor Types

    • soundtrack: Replace or generate the soundtrack for a video.
    • voice_replacement: Replace voice tracks in a video remux flow.
    • audio_edit: Process a standalone audio file in late audio postprocessing.

    Required Methods by Type

    • soundtrack: generate_soundtrack(method, video_path, prompt="", negative_prompt="", seed=-1, duration=0, output_path=None, send_cmd=None, status_callback=None, **kwargs)
    • voice_replacement: replace_voice_tracks(method, audio_tracks, voice_sample=None, voice_sample2=None, output_dir="", prefix="", process_files=None, **kwargs)
    • audio_edit: process_audio_file(method, audio_source, output_path=None, status_callback=None, **kwargs)
    class MyAudioProcessor:
    def query_audio_processor_def(self):
        return {
            "name": "MyAudioProcessor",
            "processor_types": ("soundtrack",),
            "methods": [("MyAudioProcessor", "myaudio")],
            "method_types": {"myaudio": ("soundtrack",)},
            "needs_prompt": {"myaudio": True},
            "config_key": "myaudio",
            "pos": 100,
        }

    def validate_method(self, method, **kwargs): 
        # Return "" for success or error text
        return ""

    def generate_soundtrack(self, method, video_path, prompt="", ...):
        # Return path to audio file to mux into video
        return "/path/to/output.wav"

  8. Use trained Seed-VC checkpoints for inference

    main

    After training, checkpoints and configs are stored in ./runs/<run-name>/.

    • Checkpoint file: ft_model.pth
    • Config file: A file with the same name as your training config.

    To perform inference, specify these paths and provide a reference audio file of the target speaker (similar to zero-shot usage).

  9. Register a Temporal Upsampler Plugin

    main

    You can register upsamplers via core code modification or through the plugin system.

    Via Core Code

    Add the handler class path to the temporal_upsampler_handlers list in postprocessing/temporal_upsamplers.py:

    temporal_upsampler_handlers = [
    "postprocessing.my_temporal.temporal_upsampler.MyTemporalUpsampler",
]

    Via Plugin (Recommended)

    Add temporal_upsampler_handlers to your plugin's plugin_info.json.

    • Use relative paths starting with . for paths relative to the plugin package root.
    • Use absolute import paths for other locations.

    Example plugin_info.json:

    {
  "name": "My Plugin",
  "temporal_upsampler_handlers": [
    ".temporal.MyTemporalUpsampler",
    "./nested/other_temporal.py:OtherTemporalUpsampler",
    "postprocessing.my_temporal.temporal_upsampler.MySharedTemporalUpsampler"
  ]
}

  11. Best practices for developing with the Image Editor

    main

    When extending or modifying the editor, follow these implementation guidelines:

    • Resource Management: Always call cleanup() to release textures, sprites, and event listeners to prevent memory leaks.
    • Coordinate Systems: Be mindful of the differences between global, local, and scaled coordinate systems.
    • State Management: Always update state through the provided methods to ensure Svelte stores trigger the necessary reactive updates.
    • Undo/Redo: Ensure all state-altering operations are wrapped in a Command to support the undo/redo history.