VILA: Vision Language Models

VILA supports the AutoProcessor class to prepare data for inference.

Important: You must call model.eval() before inference; otherwise, the model will remain in training mode and pad to the right.

Single Call

Use processor.apply_chat_template to format conversation dictionaries (containing role and content with type: image or type: text) into a prompt string, then pass that string to the processor.

Batch Call

Pass a list of conversation dictionaries to apply_chat_template via a list comprehension, then pass the resulting list of texts to the processor to handle multiple inputs at once.

from transformers import AutoProcessor, AutoModel

model_path = "Efficient-Large-Model/NVILA-Lite-2B-hf-preview"
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto")
model.eval() # Critical: set to eval mode

# Single call example
gpt_conv = [{
    "role": "user",
    "content": [
        {"type": "image", "path": "https://nvlabs.github.io/VILA/asset/example.jpg"},
        {"type": "text", "text": "Describe this image."}
    ]
}]
text = processor.apply_chat_template(gpt_conv, tokenize=False, add_generation_prompt=True)
inputs = processor([text])

output_ids = model.generate(
    input_ids=inputs.input_ids,
    media=inputs.media,
    media_config=inputs.media_config,
    generation_config=model.generation_config,
    max_new_tokens=256,
)
print(processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True))

# Batch call example
gpt_conv1 = [{"role": "user", "content": [{"type": "image", "path": "..."}, {"type": "text", "text": "..."}]}]
gpt_conv2 = [{"role": "user", "content": [{"type": "image", "path": "..."}, {"type": "text", "text": "..."}]}]

messages = [gpt_conv1, gpt_conv2]
texts = [processor.apply_chat_template(msg, tokenize=False, add_generation_prompt=True) for msg in messages]
inputs = processor(texts)

output_ids = model.generate(
    input_ids=inputs.input_ids,
    media=inputs.media,
    media_config=inputs.media_config,
    generation_config=model.generation_config,
    max_new_tokens=256,
)
output_texts = processor.tokenizer.batch_decode(output_ids, skip_special_tokens=True)

You can load VILA models using AutoConfig and AutoModel.from_config, or more directly via AutoModel.from_pretrained. Use trust_remote_code=True when loading.

To generate content, use the model.generate_content method. This method accepts a list containing either raw text strings or a combination of PIL.Image objects and text strings.

from transformers import AutoConfig, AutoModel
from termcolor import colored
import PIL.Image

model_path = "Efficient-Large-Model/NVILA-Lite-2B-hf-preview"

# Load model directly
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto")

# Example: Generate with raw text
res = model.generate_content([
    "how are you today?"
])
print(colored(res, "cyan", attrs=["bold"]))

# Example: Generate with text + image
response = model.generate_content([
    PIL.Image.open("inference_test/test_data/caption_meat.jpeg"),
    "describe the image?"
])
print(colored(response, "cyan", attrs=["bold"]))