Qwen3-VL Vision-Language Model

Qwen3-VL is a vision-language model series designed for advanced multimodal tasks. Key capabilities include:

• Visual Agent: Ability to operate PC/mobile GUIs by recognizing elements, understanding functions, and invoking tools.
• Visual Coding: Generation of Draw.io, HTML, CSS, and JS from visual inputs.
• Spatial Perception: 2D and 3D grounding, judging object positions, viewpoints, and occlusions.
• Long Context & Video: Native 256K context (expandable to 1M) for processing books and long-duration videos.
• Multimodal Reasoning: Strong performance in STEM and Math through causal and logical analysis.
• Advanced OCR: Support for 32 languages with robustness against low light, blur, and tilt.
• Architectures: Available in Dense and MoE (Mixture-of-Experts) formats, with both Instruct and Thinking (reasoning-enhanced) editions.

When a conversation contains multiple images or videos, you can use add_vision_id=True in apply_chat_template to automatically insert labels (e.g., "Picture 1:", "Video 1:") before the visual tokens. This helps the model better reference specific visual inputs in its response.

# add ids to help model reference multiple visual inputs
prompt_with_id = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    add_vision_id=True
)

Inference Output (JSONL)

Each line contains the question metadata, the task, and the model's result:

• result.gen: The final answer.
• result.gen_raw: The raw output (includes <think> tags for thinking models).

Evaluation Output

The evaluation produces three types of files:

1. *_eval_results.xlsx: Raw predictions with metadata.
2. *_eval_results_eval.xlsx: Detailed results including extracted answers (res), logs, and extraction flags.
3. *_eval_results_eval_score.csv: A summary of accuracy (acc) broken down by category (e.g., Algebra, Geometry).

Inference Output (.jsonl)

Each line contains:

• question_id: ID of the question
• annotation: Ground truth annotations
• extra_info: Metadata including dataset_name, img_id, img_path, and image dimensions
• result: Contains gen (JSON string of bounding boxes and labels) and gen_raw (the raw model output)
• messages: Conversation history

Evaluation Output (.json)

A JSON object containing metrics for each dataset and an overall average:

• mAP: Mean Average Precision at IoU 0.5:0.95
• mAP_50: mAP at IoU 0.5
• mAP_75: mAP at IoU 0.75
• mAP_s: mAP for small objects (area < 32²)
• mAP_m: mAP for medium objects (32² < area < 96²)
• mAP_l: mAP for large objects (area > 96²)

Inference Output (JSONL)

Each line contains:

• question_id: ID of the question
• annotation: Original question data (index, question, options, answer)
• task: Dataset name
• result: Contains gen (final answer) and gen_raw (raw output including thinking process)
• messages: Conversation history

Evaluation Output

1. CSV file (*_eval_results.csv): Detailed per-sample results including index, question, prediction, extracted_answer, extraction_method, gt (ground truth), hit, and split.
2. JSON file (*_eval_results_acc.json): Accuracy summary containing overall_accuracy and accuracy_by_split.

When passing images to the model (either via local inference or an OpenAI-compatible API), images must be converted to Base64 strings. The data URI format must include the correct MIME type.

Supported formats:

• PNG: f"data:image/png;base64,{base64_image}"
• JPEG: f"data:image/jpeg;base64,{base64_image}"
• WEBP: f"data:image/webp;base64,{base64_image}"

import base64
from io import BytesIO

def image_to_base64(img, format="PNG"):
    buffered = BytesIO()
    img.save(buffered, format=format)
    img_bytes = buffered.getvalue()
    img_base64 = base64.b64encode(img_bytes).decode('utf-8')
    return img_base64

# Usage in content list:
# "image_url": {"url": f"data:image/png;base64,{base64_image}"}

The evaluation process uses a two-stage approach: rule-based extraction followed by LLM-based extraction (using a judge like GPT-4) for ambiguous cases. To customize this logic, modify eval_utils.py:

• can_infer_option(): Modify rules for extracting options (e.g., A/B/C/D).
• can_infer_text(): Modify text matching logic.
• build_prompt(): Customize the prompt used by the LLM judge.

Qwen3-VL introduces several architectural improvements for enhanced multimodal processing:

• Interleaved-MRoPE: Uses robust positional embeddings with full-frequency allocation over time, width, and height to improve long-horizon video reasoning.
• DeepStack: Fuses multi-level ViT (Vision Transformer) features to capture fine-grained details and improve image-text alignment.
• Text–Timestamp Alignment: Enables precise, timestamp-grounded event localization for improved video temporal modeling (moving beyond T-RoPE).

Perform inference using run_realworldqa.py with the infer command. You can use standard instruct models or thinking models (which require different generation parameters).

Instruct Model Example:

python run_realworldqa.py infer \
    --model-path /path/to/Qwen3-VL-Instruct \
    --data-dir /path/to/data \
    --dataset RealWorldQA \
    --output-file results/predictions.jsonl \
    --max-new-tokens 32768 \
    --temperature 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --repetition-penalty 1.0 \
    --presence-penalty 1.5

Thinking Model Parameters: For models like Qwen3-VL-2B-Thinking, use these adjusted parameters for better reasoning:

• --max-new-tokens 32768
• --temperature 0.6
• --top-p 0.95
• --top-k 20
• --repetition-penalty 1.0
• --presence-penalty 0.0

Use the official qwenllm/qwenvl Docker image to simplify deployment. You will need to have the NVIDIA driver installed and download the model files to your local environment.

docker run --gpus all --ipc=host --network=host --rm --name qwen3vl -it qwenllm/qwenvl:qwen3vl-cu128 bash