repository·main·Indexed 23 days ago
https://github.com/qwenlm/qwen3-vl-embeddingOfficial implementation of Qwen3-VL-Embedding and Qwen3-VL-Reranker, multimodal models built on the Qwen3-VL foundation. The series supports text, images, screenshots, and videos for information retrieval and cross-modal understanding. It features a two-stage pipeline: a Dual-Tower embedding model for efficient initial recall and a Single-Tower reranker for precise relevance scoring. Available in 2B and 8B parameter versions, with support for vLLM accelerated inference.
| Feature | Qwen3-VL-Embedding | Qwen3-VL-Reranker |
|---|---|---|
| Core Function | Semantic Representation, Embedding Generation | Relevance Scoring, Pointwise Re-ranking |
| Input | Single modality or mixed modalities | (Query, Document) pair with single- or mixed-modal inputs |
| Architecture | Dual-Tower | Single-Tower |
| Mechanism | Efficient Retrieval | Deep Inter-Modal Interaction, Precise Alignment |
| Output | Semantic Vector | Relevance Score |
The Qwen3-VL model series is designed for multimodal information retrieval using a two-stage pipeline:
Qwen3-VL-Embedding model (Dual-Tower architecture) generates semantically rich vectors from single or mixed-modal inputs (text, images, screenshots, videos). These vectors are used for efficient initial retrieval.Qwen3-VL-Reranker model (Single-Tower architecture) takes (Query, Document) pairs and performs pointwise reranking. It uses cross-attention for deep inter-modal interaction and outputs a precise relevance score by predicting the generation probability of special tokens (yes and no).To download models from ModelScope, install modelscope using uv and use the modelscope CLI.
Example for Qwen3-VL-Embedding-2B:
uv pip install modelscope
modelscope download --model qwen/Qwen3-VL-Embedding-2B --local_dir ./models/Qwen3-VL-Embedding-2Buv pip install modelscope
modelscope download --model qwen/Qwen3-VL-Embedding-2B --local_dir ./models/Qwen3-VL-Embedding-2BTo generate embeddings for text, images, or multimodal inputs using the Qwen3VLEmbedder class, provide a list of dictionaries where each dictionary represents a multimodal object. You can include text, image, and video keys. For text-only tasks, you can optionally include an instruction to guide the embedding representation.
Supported input types for image and video include local file paths, URLs, or PIL.Image.Image instances. All input types support both single objects and lists of objects.
import torch
from src.models.qwen3_vl_embedding import Qwen3VLEmbedder
model = Qwen3VLEmbedder(
model_name_or_path="./models/Qwen3-VL-Embedding-2B",
# flash_attention_2 for better acceleration and memory saving
# torch_dtype=torch.bfloat16,
# attn_implementation="flash_attention_2"
)
inputs = [{
"text": "A woman playing with her dog on a beach at sunset.",
"instruction": "Retrieve images or text relevant to the user's query.",
}, {
"text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust."
}, {
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}, {
"text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
}]
embeddings = model.process(inputs)
print(embeddings @ embeddings.T)The models support vLLM for accelerated inference.
Requirements:
vLLM >= 0.14.0.Usage:
examples/embedding_vllm.ipynb.examples/reranker_vllm.ipynb.To reproduce the evaluation results for the Embedding Model on the MMEB v2 benchmark, follow these steps:
Download the evaluation data:
bash data/evaluation/mmeb_v2/download_data.shRun the evaluation script:
bash scripts/evaluation/mmeb_v2/eval_embedding.shRunning the script without arguments will display the required parameters. The script automatically evaluates tasks and collects results.
bash data/evaluation/mmeb_v2/download_data.sh
bash scripts/evaluation/mmeb_v2/eval_embedding.shTo install the project, clone the repository and run the provided setup script which uses uv to manage dependencies.
# Clone the repository
git clone https://github.com/QwenLM/Qwen3-VL-Embedding.git
cd Qwen3-VL-Embedding
# Run the script to setup the environment
bash scripts/setup_environment.sh
# Activate the environment
source .venv/bin/activategit clone https://github.com/QwenLM/Qwen3-VL-Embedding.git
cd Qwen3-VL-Embedding
bash scripts/setup_environment.sh
source .venv/bin/activateTo download models from Hugging Face, install huggingface-hub using uv and use the huggingface-cli.
Example for Qwen3-VL-Embedding-2B:
uv pip install huggingface-hub
huggingface-cli download Qwen/Qwen3-VL-Embedding-2B --local-dir ./models/Qwen3-VL-Embedding-2Buv pip install huggingface-hub
huggingface-cli download Qwen/Qwen3-VL-Embedding-2B --local-dir ./models/Qwen3-VL-Embedding-2BTo perform reranking tasks, use the Qwen3VLReranker class. The input must be a dictionary containing a query (a multimodal object) and a list of documents (a list of multimodal objects). You can also provide an optional instruction for the task and video sampling settings (fps, max_frames).
import torch
from src.models.qwen3_vl_reranker import Qwen3VLReranker
model = Qwen3VLReranker(
model_name_or_path="./models/Qwen3-VL-Reranker-2B",
# flash_attention_2 for better acceleration and memory saving
# torch_dtype=torch.bfloat16,
# attn_implementation="flash_attention_2"
)
inputs = {
"instruction": "Retrieve images or text relevant to the user's query.",
"query": {"text": "A woman playing with her dog on a beach at sunset."},
"documents": [
{"text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust."},
{"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
{"text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.",
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"}
],
"fps": 1.0,
"max_frames": 64
}
scores = model.process(inputs)
print(scores)To reproduce the evaluation results for the Reranking Model on the MMEB v2 retrieval split, follow these steps:
Download the evaluation data:
bash data/evaluation/mmeb_v2/download_data.shRun the evaluation script:
bash scripts/evaluation/mmeb_v2/eval_reranker.shRunning the script without arguments will display the required parameters. The script automatically evaluates tasks and collects results.
bash data/evaluation/mmeb_v2/download_data.sh
bash scripts/evaluation/mmeb_v2/eval_reranker.shThe reranker accepts multimodal inputs (text and/or images) for both the query and the candidate documents.
Input structure:
instruction: (Optional) A string describing the task.query: A dictionary containing 'text' and/or 'image'.documents: A list of dictionaries, where each dictionary contains 'text' and/or 'image'.Images can be provided as:
http:// or https://).The eval_reranker.py script is a distributed evaluation tool for reranker models. It takes TopK retrieval results (generated by an embedding model) and uses a Qwen3VLReranker to re-score and re-order them.
To run the evaluation, use the script with arguments parsed from three main dataclasses: RerankArguments, DataArguments, and EvalArguments. The script supports distributed evaluation using PyTorch's dist (NCCL backend) and expects a dataset configuration YAML file.
Workflow:
Qwen3VLReranker (supports flash_attention_2 and bfloat16).*_pred.jsonl files from the embedding evaluation output directory to get the initial TopK candidate IDs.hit, ndcg, precision, recall, f1, map, mrr) using RankingMetrics..jsonl file containing the reranked predictions.