DOCSEARCHbyAlgolia
Submit Docs

isi-dev Google Colab Notebooks

repository·main·Indexed 19 days ago

https://github.com/isi-dev/google-colab_notebooks

A collection of Google Colab notebooks for AI video and audio generation. Includes setup and workflows for LatentSync high-fidelity lip-syncing, Wan2.1 VACE and 14B Image-to-Video (I2V) using ComfyUI and GGUF models, Minimax-H3 video generation, and dots.tts-soar text-to-speech synthesis. Features include Practical-RIFE frame interpolation, CausVid LoRA integration, and VRAM management optimizations for T4 GPUs.

Tokens
13.8K
Snippets
38
Records
54
Agent score
67%

What's inside isi-dev/google-colab_notebooks

  1. How to use Qwen-Image-Edit prompting and masking

    main

    The Qwen-Image-Edit model (specifically the 2511 version) is designed for prompt-based editing. While it supports inpainting via a mask editor, prompt-only editing generally yields higher quality results as it aligns with the model's original training.

    Prompting Strategy: Use clear, descriptive prompts that reference multiple uploaded images. For example, if you upload three images, use a prompt like: "Let the girl in image 1 wear the outfit from image 2 and assume the pose from image 3."

    Inpainting vs. Prompting:

    • Prompt-only: Generally produces more impressive results.
    • Mask Editor (Inpainting): Offers more spatial control but can sometimes produce less impressive results or fail to apply changes effectively.

  2. Apply LoRA and Speedup LoRA to the Model

    main

    You can enhance the model using LoRAs or use a specific Turbo LoRA for faster generation.

    • Speedup LoRA: Enabling use_turbo_lora loads a specific LoRA that reduces the required steps to 4 and sets cfg to 1.
    • Standard LoRAs: You can apply up to four different LoRAs (use_lora, use_lora2, etc.) by providing the LoRA path and a corresponding LoRA_Strength.

  3. Optimize Generation Speed with Teacache and LoRAs

    main

    Generation speed can be significantly improved using Teacache or specific LoRAs.

    Teacache Settings: Adjust rel_l1_thresh to trade quality for speed. A value of 0.275 can reduce generation time from ~33 minutes to ~18 minutes for a 480x480 video with minimal quality loss.

    Speed-up LoRAs and Recommended Settings: If using these LoRAs, it is recommended to set rel_l1_thresh to zero.

    LoRA Typecfg_scalestepssampler_nameschedulerflow_shiftstrength
    causvid14uni_pcsimple50.8
    lightx2v14LCMsimple81
    fusionx16uni_pcsimple51

    Note: fusionx already contains the causvid LoRA, but you can experiment with combinations.

  4. Choose a HiDream model version

    main

    The HiDream_fp8 implementation supports three distinct model versions, which affect generation speed, quality, and resource usage. You should select one during the environment setup phase:

    • fast: Optimized for speed. Recommended settings: steps=16, shift=3.0.
    • dev: Balanced version. Recommended settings: steps=28, shift=6.0.
    • full: Highest quality. Recommended settings: steps=50, cfg_scale=5.0, shift=3.0.

  5. Optimize Generation Speed with Teacache

    main

    Teacache can significantly reduce video generation time with minimal quality loss. It is controlled via the rel_l1_thresh parameter in the generate_video function.

    • Speed vs. Quality: Increasing rel_l1_thresh results in faster generation but higher quality loss.
    • Benchmark: For a 480x480, 20-step, 65-frame generation on a T4 GPU, using rel_l1_thresh = 0.275 reduced generation time from ~33 minutes to ~18 minutes.

  6. Prepare Environment for Faster WAN 2.1

    main

    To use the Faster WAN 2.1 Image-to-Video notebook, you must first upgrade the core PyTorch libraries to ensure compatibility with CUDA 11.8. Run the following command in a cell to install the required versions of torch, torchvision, and torchaudio.

    #!pip install --upgrade --quiet torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

  7. Configure dots.tts-soar Studio interface modes

    main

    The notebook provides two ways to interact with the model:

    1. Colab Native Form: Uses Colab's built-in form widgets. This mode supports a checkbox upload_reference_audio which triggers the standard Colab file uploader to provide a reference audio for voice cloning.
    2. Gradio Web UI: Launches a Gradio dashboard within the notebook for a more traditional web-app experience. This is useful for interactive testing and sharing via a public URL.

  8. Download Qwen-Image Models

    main

    The notebook uses several model components. You can specify the download URLs for the following components:

    • Qwen Model (UNET): The core model (e.g., GGUF format).
    • Text Encoder: The CLIP/T5 model used for text encoding.
    • VAE: The Variational Autoencoder for encoding/decoding latents.
    • Speedup LoRA: A LoRA that enables faster generation by setting steps to 4 and CFG to 1.

    Models can be sourced from HuggingFace (Comfy-Org, QuantStack, or unsloth) or downloaded via aria2c for faster speeds.

  9. Setup the Wan2.1 VACE Environment in Google Colab

    main

    To run the Wan2.1 VACE workflow, you must first prepare the environment by installing ComfyUI, GGUF support, and the Practical-RIFE interpolation models. This setup is optimized for the T4 GPU.

    Key Installation Steps:

    1. Install torch==2.6.0 and torchvision==0.21.0.
    2. Clone ComfyUI (version v0.3.36) and the ComfyUI_GGUF custom node.
    3. Install Practical-RIFE and download necessary pre-trained interpolation models (IFNet_HDv3.py, RIFE_HDv3.py, etc.) from HuggingFace.
    4. Install required Python packages including torchsde, diffusers, transformers, xformers==0.0.29.post2, and einops.
    # Run this in a Colab cell to initialize the environment
!pip install torch==2.6.0 torchvision==0.21.0
%cd /content
!git clone --branch ComfyUI_v0.3.36 https://github.com/Isi-dev/ComfyUI
%cd /content/ComfyUI/custom_nodes
!git clone https://github.com/Isi-dev/ComfyUI_GGUF.git
%cd /content/ComfyUI/custom_nodes/ComfyUI_GGUF
!pip install -r requirements.txt

  10. Prepare Environment for Qwen-Image-Edit

    main

    To run the Qwen-Image-Edit notebook, you must install specific versions of PyTorch and several ComfyUI custom nodes. The environment setup involves cloning ComfyUI, installing the ComfyUI_GGUF and comfyui_controlnet_aux nodes, and installing a variety of Python dependencies including diffusers, accelerate, and segment_anything.

    Key Dependencies:

    • torch==2.8.0 and torchvision==0.23.0 (or specific CUDA versions via index-url)
    • ComfyUI (version v0.3.60 is explicitly cloned)
    • ComfyUI_GGUF for GGUF model support
    • comfyui_controlnet_aux for pose estimation
    • ComfyUI_Img2PaintingAssistant

    Note: The setup script uses aria2 for optimized downloads and installs several specialized packages like sageattention and triton.

    !pip install torch==2.8.0 torchvision==0.23.0
%cd /content
!git clone --branch ComfyUI_v0.3.60 https://github.com/Isi-dev/ComfyUI
%cd /content/ComfyUI/custom_nodes
!git clone https://github.com/Isi-dev/ComfyUI_Img2PaintingAssistant
!git clone --branch forQwen https://github.com/Isi-dev/ComfyUI_GGUF.git
!git clone https://github.com/Isi-dev/comfyui_controlnet_aux
%cd /content/ComfyUI/custom_nodes/ComfyUI_GGUF
!pip install -r requirements.txt

  11. Setup the Sonic environment in Google Colab

    main

    To use Sonic for portrait animation, you must set up a ComfyUI environment with specific custom nodes and model weights. The setup involves cloning the ComfyUI repository, the ComfyUI_Sonic custom nodes, installing several Python dependencies (including torchsde, diffusers, and transformers), and downloading required model weights using aria2c.

    Hardware Note:

    • T4 GPU: Very slow. Tested up to 5 seconds of audio (approx. 19 minutes for 5s).
    • A100 GPU: Much faster. Tested up to 45 seconds of audio (approx. 8 minutes for 17s).

    Required Models: Models are downloaded to /content/ComfyUI/models/sonic and /content/ComfyUI/models/checkpoints:

    • audio2bucket.pth
    • audio2token.pth
    • unet.pth
    • yoloface_v5m.pt
    • RIFE/flownet.pkl
    • whisper-tiny configs/models
    • svd_xt_1_1.safetensors (SVD checkpoint)
    # Run these in a Colab cell to initialize the environment
%cd /content/
!git clone https://github.com/Isi-dev/ComfyUI
%cd /content/ComfyUI/custom_nodes
!git clone https://github.com/Isi-dev/ComfyUI_Sonic
%cd /content/ComfyUI

# Installation of dependencies and models is handled by the notebook's setup cells

  12. Setup Environment for HiDream_fp8

    main

    To use HiDream_fp8 for text-to-image generation in Google Colab, you must first set up the environment by installing specific Python packages and cloning the ComfyUI repository.

    Important Considerations:

    • Model Versions: Choose between fast, dev, or full versions. Each model is approximately 17.1GB, so ensure you have sufficient compute units (e.g., Colab Pro).
    • Dependencies: The setup installs torch, torchvision, torchsde, einops, diffusers, accelerate, and xformers. It also uses aria2 for optimized model downloads.

    Follow the notebook's setup cell to clone ComfyUI and install all required dependencies automatically.

    # Run the setup cell in the notebook to:
# 1. Clone ComfyUI
# 2. Install pip packages (torch, diffusers, xformers, etc.)
# 3. Install apt packages (aria2)
# 4. Download the selected HiDream model and required text encoders/VAE