VideoLLaMA 3 Documentation

Training data should be organized under a data_root directory. Annotations should be provided in .jsonl format (recommended for memory efficiency) containing a list of dictionaries. Each dictionary represents a conversation involving either an image or a video.

Annotation Format:

• For images: {"image": ["path/to/image.jpg"], "conversations": [...]}
• For videos: {"video": ["path/to/video.mp4"], "conversations": [...]}

Conversation Format: Conversations consist of alternating human and gpt roles. Use <image> or <video> tokens within the human value to indicate the media placement.

[
    {
        "image": ["images/xxx.jpg"],
        "conversations": [
            {
                "from": "human",
                "value": "<image>\nWhat are the colors of the bus in the image?"
            },
            {
                "from": "gpt",
                "value": "The bus in the image is white and red."
            }
        ]
    },
    {
        "video": ["videos/xxx.mp4"],
        "conversations": [
            {
                "from": "human",
                "value": "<video>\nWhat are the main activities that take place in the video?"
            },
            {
                "from": "gpt",
                "value": "The main activities..."
            }
        ]
    }
]

To set up VideoLLaMA 3 for stable inference, ensure you have Python >= 3.10 and CUDA >= 11.8. Follow these installation steps to install specific compatible versions of PyTorch, Flash-attn, Transformers, and video processing dependencies.

Important Compatibility Note: If you are using CUDA 11.8 with torch==2.4.0 and torchvision==0.19.0, you must use flash-attn==2.7.3. If using different Python or CUDA versions, consult the flash-attn releases for compatible wheels to avoid breaking the setup.

# PyTorch and torchvision for CUDA 11.8
pip install torch==2.4.0 torchvision==0.19.0 --extra-index-url https://download.pytorch.org/whl/cu118

# Flash-attn pinned to a compatible version
pip install flash-attn==2.7.3 --no-build-isolation --upgrade

# Transformers and accelerate
pip install transformers==4.46.3 accelerate==1.0.1

# Video processing dependencies
pip install decord ffmpeg-python imageio opencv-python

Visual Grounding is the task of identifying an object from a text prompt and outputting its bounding box in the format [[x0, y0, x1, y1]].

Workflow:

1. Construct a conversation list containing a user role with an image type and a text type prompt (e.g., "Where is the [object]? Answer in [[x0,y0,x1,y1]] format.").
2. Process the conversation using processor(conversation=conversation, return_tensors="pt").
3. Move inputs to GPU and ensure pixel_values are in torch.bfloat16.
4. Generate output using model.generate(**inputs).
5. Decode the response using processor.batch_decode().
6. Parse the bounding box from the text response using a helper like extract_boxes().

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image", 
                "image": {"image_path": image_path}
            },
            {
                "type": "text", 
                "text": "Where is the white car? Answer in [[x0,y0,x1,y1]] format.",
            },
        ]
    }
]

# Single-turn conversation
inputs = processor(conversation=conversation, return_tensors="pt")
inputs = {k: v.cuda() if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=128)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(response)

Visual Referring allows the model to answer questions about a specific region of an image provided via a bounding box.

Workflow:

1. Define the target region using a raw bounding box [x0, y0, x1, y1].
2. Convert the raw box to a normalized format (0-1000 scale) using raw2normalized(box, h, w).
3. Construct a conversation where the text prompt includes the normalized box coordinates (e.g., f"What is the license plate number of {normalized_box}?").
4. Process and generate as with standard inference.

input_box = [150,50,530,300]
normalized_box = raw2normalized(input_box, image.size[1], image.size[0])

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image", 
                "image": {"image_path": image_path}
            },
            {
                "type": "text", 
                "text": f"What is the license plate number of {normalized_box}?"
            },
        ]
    }
]

# Single-turn conversation
inputs = processor(conversation=conversation, return_tensors="pt")
inputs = {k: v.cuda() if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
if "pixel_values" in inputs:
    inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)

output_ids = model.generate(**inputs, max_new_tokens=128)
response = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(response)