DOCSEARCHbyAlgolia
Submit Docs

ViperGPT Documentation

repository·main·Indexed 23 days ago

https://github.com/cvlab-columbia/viper

ViperGPT is a visual inference system that utilizes Large Language Models to generate and execute Python code for reasoning about images and videos. It features a modular system for vision models, YAML-based configuration via OmegaConf, and support for batch inference on datasets. The system includes a modified GLIP module and integrates with various pretrained models to perform complex visual reasoning tasks.

Tokens
1.6K
Snippets
3
Records
8
Agent score
33%

What's inside ViperGPT

  1. How to implement a new Vision Model

    main

    ViperGPT uses a modular system for vision models. To add a new model, you must implement a new class in vision_models.py that follows these requirements:

    1. Inherit from BaseModel.
    2. Implement the forward method.
    3. Implement the name method (this string is used to call the model via vision_processes.py).

    vision_processes.py will automatically detect new models implemented in vision_models.py and manage their lifecycle in a multiprocessing environment.

  2. Quickstart Installation

    main

    To quickly set up ViperGPT, clone the repository recursively and run the setup scripts. Ensure you have CUDA installed and the vipergpt conda environment active before running the GLIP installation.

    Warning: ViperGPT executes code generated by an LLM. This can be dangerous. It is recommended to run in a sandboxed environment. By default, execute_code is set to False in the config to prevent automatic execution.

    # Clone the repository
git clone --recurse-submodules https://github.com/cvlab-columbia/viper.git

# Setup
cd viper
export PATH=/usr/local/cuda/bin:$PATH
bash setup.sh

# Install modified GLIP
cd GLIP
python setup.py clean --all build develop --user
cd ..

# Set up OpenAI key
echo YOUR_OPENAI_API_KEY_HERE > api.key

  3. Run ViperGPT on a Dataset

    main

    To run inference on a batch of queries rather than a single image, use main_batch.py.

    Dataset Format: Your dataset directory must contain a queries.csv file and images/ or videos/ subdirectories. The queries.csv should follow this format: query,answer,image_name/video_name (the answer field is optional and used for evaluation).

    Execution: Use the CONFIG_NAMES environment variable to specify which configuration file(s) to use.

  4. Detailed Installation and Dependencies

    main

    ViperGPT requires a specific installation sequence to ensure compatibility between CUDA kernels and PyTorch versions, particularly for the modified GLIP module.

    1. Clone with submodules: git clone --recurse-submodules https://github.com/cvlab-columbia/viper.git
    2. Environment Setup: Use bash setup_env.sh to create a conda environment.
    3. GLIP Installation: You must install the provided version of GLIP from the local directory to ensure the updated CUDA kernels work with newer PyTorch versions.
    4. Pretrained Models: While many models download automatically, some require manual download using download_models.sh. Store them in the directory specified by path_pretrained_models in your config (default is ./pretrained_models/).
    5. OpenAI Key: Create a file named api.key in the root directory containing your OpenAI API key.
    export PATH=/usr/local/cuda/bin:$PATH
bash setup_env.sh
conda activate vipergpt
cd GLIP
python setup.py clean --all build develop --user
cd ..

  5. Configure ViperGPT via YAML

    main

    Configuration is managed through YAML files located in the configs/ directory. The base configuration is defined in configs/base_config.yaml.

    To run a custom configuration:

    1. Create a new file (e.g., configs/my_config.yaml) that inherits from or overwrites base_config.yaml.
    2. Update the following key parameters:
      • path_pretrained_models: Path to the directory containing manually downloaded models.
      • dataset.data_path: Path to your dataset directory.
      • execute_code: Set to True to allow automatic execution of generated code (use with caution).
      • multiprocessing: Set to True to enable parallel processing for both samples and models.

    Multiple configuration files can be passed via the command line and will be merged in the order provided.

  6. Quick Start with ViperGPT

    main

    To perform visual reasoning using ViperGPT, follow these steps:

    1. Configure Parameters: Before running the code, modify any necessary parameters in configs/my_config.yaml. For example, if you encounter GPU memory issues, you can downgrade the BLIP model type by changing blip_v2_model_type from XXL to XL.
    2. Load Image and Query: Use load_image(url_or_path) to load your visual input and define your reasoning query as a string.
    3. Generate Code: Call get_code(query) to generate the Python code required to solve the visual reasoning task.
    4. Execute: Use execute_code(code, im, show_intermediate_steps=True) to run the generated code on the image. Setting show_intermediate_steps=True allows you to see the step-by-step execution process.
    from main_simple_lib import *

im = load_image('https://viper.cs.columbia.edu/static/images/kids_muffins.jpg')
query = 'How many muffins can each kid have for it to be fair?'

show_single_image(im)
code = get_code(query)
execute_code(code, im, show_intermediate_steps=True)

  7. Configure ViperGPT via YAML

    main

    ViperGPT parameters are managed through a YAML configuration file (e.g., configs/my_config.yaml).

    One common adjustment is the BLIP model size. If your GPU lacks sufficient memory, locate the blip_v2_model_type key and change its value from XXL to XL.

  8. Reference: ViperGPT Code Structure

    main

    Overview of the core components:

    ComponentDescription
    vision_models.pyContains BaseModel subclasses for pretrained models
    vision_processes.pyBridge between models and the system; manages multiprocessing
    main_batch.pyEntry point for running inference on entire datasets
    main_simple.ipynbJupyter notebook for single image/video debugging and exploration
    image_patch.py / video_segment.pyClasses representing data units that call the vision processes
    configs/YAML configuration files (using OmegaConf)
    datasets/torch.utils.data.Dataset implementations
    prompts/Prompt templates for LLM interaction
    utils.pyAuxiliary functions and utilities