DOCSEARCHbyAlgolia
Submit Docs

OmniLottie

repository·main·Indexed 20 days ago

https://github.com/openvglab/omnilottie

A multimodal Lottie generator that utilizes Vision-Language Models (VLMs) to create complex vector animations from text, images, or video instructions. It supports multiple model formats, including HuggingFace and an original PyTorch format, and provides a Gradio interface for interactive generation. The project also includes MMLottieBench, a standardized evaluation dataset containing 900 samples across real and synthetic splits for text-to-Lottie, text-image-to-Lottie, and video-to-Lottie tasks.

Tokens
3.9K
Snippets
13
Records
16
Agent score
23%

What's inside OmniLottie

  1. Overview of MMLottieBench splits and tasks

    main

    MMLottieBench contains 900 samples divided into two main splits, each containing three specific task types:

    Splits

    • Real split: 450 real-world Lottie animations.
    • Synthetic split: 450 synthetically generated samples.

    Task Types (150 samples per split)

    • text2lottie: Text-to-Lottie generation.
    • text_image2lottie: Text-Image-to-Lottie generation.
    • video2lottie: Video-to-Lottie generation.

  2. How model formats differ in OmniLottie

    main

    OmniLottie supports two distinct model formats. Both produce identical results, so choose based on your existing files:

    1. 🤗 HuggingFace Format (Recommended):

      • Uses inference_hf.py and app_hf.py.
      • Files: model-*.safetensors and config.json.
      • Supports the from_pretrained() API for automatic downloading from the HuggingFace Hub.

    2. Original Format:

      • Uses inference.py and app.py.
      • File: pytorch_model.bin.
      • Intended for users who downloaded the model before HuggingFace format support was added.

  3. Download the MMLottieBench dataset

    main

    MMLottieBench is a standardized evaluation dataset for Lottie generation models. You can download it using one of three methods:

    1. Download script: Run the provided Python script.
    2. Hugging Face CLI: Use the huggingface-cli to download the dataset directly from Hugging Face.
    3. Python Code: Use the datasets library to load it automatically in your script.
    # Option 1: Using download script
python download_mmlottie_bench.py --output_dir /PATH/TO/mmlottie_bench

# Option 2: Using Hugging Face CLI
huggingface-cli download OmniLottie/MMLottieBench --repo-type dataset --local-dir /PATH/TO/mmlottie_bench
    # Option 3: Automatic download (in code)
from datasets import load_dataset
dataset = load_dataset("OmniLottie/MMLottieBench")

  4. Download OmniLottie model weights

    main

    You can download the model weights using the Hugging Face CLI. It is recommended to use the HuggingFace format (which includes config.json and .safetensors files) for new users.

    First, install the CLI tool:

    pip install huggingface-hub

    Then, download the model to a local directory:

    huggingface-cli download OmniLottie/OmniLottie --local-dir /PATH/TO/OmniLottie
    pip install huggingface-hub
huggingface-cli download OmniLottie/OmniLottie --local-dir /PATH/TO/OmniLottie

  5. Install OmniLottie

    main

    To set up an environment for OmniLottie inference, follow these steps:

    1. Clone the repository:

      git clone https://github.com/OpenVGLab/OmniLottie
cd OmniLottie

    2. Create a Conda environment with Python 3.10:

      conda create -n omnilottie python=3.10
conda activate omnilottie

    3. Install PyTorch with CUDA 12.1 support (tested environment):

      pip install torch==2.3.0+cu121 torchvision==0.18.0+cu121 --index-url https://download.pytorch.org/whl/cu121

    4. Install remaining dependencies:

      pip install -r requirements.txt
    git clone https://github.com/OpenVGLab/OmniLottie
cd OmniLottie
conda create -n omnilottie python=3.10
conda activate omnilottie
pip install torch==2.3.0+cu121 torchvision==0.18.0+cu121 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

  6. Run the Omnilottie Gradio interface

    main

    The application can be launched as a Gradio web interface. By default, it runs on 0.0.0.0:7860. Note that the demo is configured to process one request at a time; subsequent requests will be queued until the current generation completes.

    if __name__ == "__main__":
    demo = create_gradio_interface()
    demo.launch(
        server_name="0.0.0.0",
        server_port=7860,
        share=False,
        show_error=True
    )

  7. Launch the OmniLottie Gradio demo

    main

    You can run an interactive Gradio interface locally using either the HuggingFace or Original model formats.

    Using HuggingFace format (Recommended): Set the MODEL_PATH environment variable before running app_hf.py.

    # Using a local model path
MODEL_PATH=/PATH/TO/OmniLottie python app_hf.py

# Using the HF Hub (automatic download)
MODEL_PATH=OmniLottie/OmniLottie python app_hf.py

    Using Original format:

    python app.py
    MODEL_PATH=OmniLottie/OmniLottie python app_hf.py

  8. Run inference using HuggingFace format

    main

    If you are using the recommended HuggingFace format (safetensors + config.json), use inference_hf.py. You can either point to a local path or use the HuggingFace Hub ID for automatic downloading.

    Text-to-Lottie:

    python inference_hf.py --model_path OmniLottie/OmniLottie --text "A bouncing ball" --output output.json

    Image-to-Lottie:

    python inference_hf.py --model_path OmniLottie/OmniLottie --image image.png --text "rotating animation" --output output.json

    Video-to-Lottie:

    python inference_hf.py --model_path OmniLottie/OmniLottie --video video.mp4 --text "rotating animation" --output output.json

    Using a local model path:

    python inference_hf.py --model_path /PATH/TO/OmniLottie --text "A spinning star" --output output.json
    python inference_hf.py --model_path OmniLottie/OmniLottie --text "A bouncing ball" --output output.json

  9. Run inference using the Original format

    main

    If you have the pytorch_model.bin format, use inference.py. Note that the --sketch_weight flag is used to specify the model path.

    Text-to-Lottie (Single prompt):

    python inference.py --sketch_weight /PATH/TO/OmniLottie --single_text "A red ball..." --output_dir ./output_text

    Text-to-Lottie (Batch from file):

    python inference.py --sketch_weight /PATH/TO/OmniLottie --batch_text_file example/demo.txt --output_dir ./output_text

    Image-to-Lottie:

    python inference.py --sketch_weight /PATH/TO/OmniLottie --single_image /path/to/image.png --output_dir ./output_image

    Video-to-Lottie:

    python inference.py --sketch_weight /PATH/TO/OmniLottie --single_video /path/to/video.mp4 --output_dir ./output_video
    python inference.py --sketch_weight /PATH/TO/OmniLottie --single_text "A red ball..." --output_dir ./output_text

  10. Configure advanced inference parameters

    main

    When using inference.py (Original format), you can fine-tune the generation process using several flags:

    • --use_sampling: Enables sampling.
    • --temperature <float>: Controls randomness.
    • --top_p <float>: Nucleus sampling threshold.
    • --top_k <int>: Top-k sampling threshold.
    • --repetition_penalty <float>: Penalty for repeating tokens.
    • --num_candidates <int>: Number of candidates for Best-of-N selection.
    • --tokenizer_name <path>: Specify a custom tokenizer path (e.g., Qwen2.5-VL-3B-Instruct).
    • --maxlen <int>: Maximum token length.
    • --text_len <int>: Text length limit.

    Example with sampling and Best-of-N:

    python inference.py \
    --sketch_weight /PATH/TO/OmniLottie \
    --single_text "a light blue piggy bank..." \
    --use_sampling \
    --temperature 0.8 \
    --top_p 0.25 \
    --top_k 5 \
    --repetition_penalty 1.01 \
    --num_candidates 8 \
    --output_dir ./output
    python inference.py \
    --sketch_weight /PATH/TO/OmniLottie \
    --single_text "a light blue piggy bank..." \
    --use_sampling \
    --temperature 0.8 \
    --top_p 0.25 \
    --top_k 5 \
    --repetition_penalty 1.01 \
    --num_candidates 8 \
    --output_dir ./output

  11. Run benchmark inference with inference.py

    main

    Use the inference.py script to evaluate models against the MMLottieBench dataset. You can specify the split (real or synthetic), target specific task types, and limit the number of samples processed.

    CLI Arguments

    • --sketch_weight: Path to the OmniLottie model weights.
    • --mmlottie_bench_dir: Path to the downloaded MMLottieBench directory.
    • --split: Choose between real or synthetic.
    • --mmlottie_task: (Optional) Filter by task: text2lottie, text_image2lottie, or video2lottie.
    • --output_dir: Directory where results will be saved.
    • --max_samples: (Optional) Limit the number of samples processed.
    • --shuffle: (Optional) Shuffle the samples before processing.
    # Test on the real split (all tasks)
python inference.py \
    --sketch_weight /PATH/TO/OmniLottie \
    --mmlottie_bench_dir /PATH/TO/mmlottie_bench \
    --split real \
    --output_dir ./benchmark_results_real

# Test a specific task (text2lottie) on the real split
python inference.py \
    --sketch_weight /PATH/TO/OmniLottie \
    --mmlottie_bench_dir /PATH/TO/mmlottie_bench \
    --split real \
    --mmlottie_task text2lottie \
    --output_dir ./benchmark_results

# Process a limited, shuffled subset of 50 samples
python inference.py \
    --sketch_weight /PATH/TO/OmniLottie \
    --mmlottie_bench_dir /PATH/TO/mmlottie_bench \
    --split real \
    --max_samples 50 \
    --shuffle \
    --output_dir ./benchmark_results

  12. Generate Lottie animations from text

    main

    Use process_text_to_lottie to generate a Lottie animation based on a text description. This function handles model loading, inference, JSON reconstruction, and returns a previewable HTML string and a path to the generated JSON file.

    Parameters:

    • text_prompt (str): A description of the animation (e.g., "a blue bird appearing, pulsing while sliding downward").
    • max_tokens (int): Controls complexity. Higher values allow for more complex animations but take longer.
    • use_sampling (bool): Whether to use sampling during generation.
    • temperature (float): Controls diversity (higher = more diverse).
    • top_p (float): Nucleus sampling parameter.
    • top_k (int): Top-k sampling parameter.

    Returns:

    • html (str): An iframe containing the Lottie animation preview.
    • status (str): A status message containing token count, layer count, FPS, and elapsed time.
    • temp_path (str): The file path to the temporary .json file containing the Lottie data.
    html, status, temp_path = process_text_to_lottie(
    text_prompt="a red ball bouncing",
    max_tokens=5556,
    use_sampling=True,
    temperature=0.9,
    top_p=0.25,
    top_k=5
)