DOCSEARCHbyAlgolia
Submit Docs

Qwen3 Embedding

repository·main·Indexed 24 days ago

https://github.com/qwenlm/qwen3-embedding

A series of high-performance text embedding and reranking models (0.6B, 4B, and 8B) supporting over 100 languages. Designed for retrieval, classification, clustering, and bitext mining, these models feature Matryoshka Representation Learning (MRL) for custom dimensions and are instruction-aware. The documentation provides implementation guides for using the models with Transformers, vLLM, and Sentence Transformers, as well as evaluation scripts for MTEB benchmarks and the RAR-b benchmark.

Tokens
8.1K
Snippets
18
Records
25
Agent score
84%

What's inside Qwen3-Embedding

  1. Overview of Qwen3 Embedding series

    main

    The Qwen3 Embedding series is a collection of text embedding and reranking models designed for tasks such as text retrieval, code retrieval, classification, clustering, and bitext mining. The series includes models of various sizes (0.6B, 4B, and 8B) and inherits the multilingual and long-text understanding capabilities of the Qwen3 foundational models.

    Key features include:

    • Versatility: High performance on MTEB multilingual leaderboards.
    • Flexibility: Supports custom embedding dimensions via Matryoshka Representation Learning (MRL) and user-defined instructions.
    • Multilingualism: Supports over 100 languages and various programming languages.

  2. Configure Reranker training sampling and loss

    main

    Reranker training is controlled by the following environment variables and loss types:

    Sampling Control:

    • MAX_POSITIVE_SAMPLES: Max positive examples per query (default: 1).
    • MAX_NEGATIVE_SAMPLES: Max negative examples per query (default: 7).

    Loss Types:

    1. Pointwise Loss (--loss_type generative_reranker):

      • Uses Binary cross-entropy.
      • GENERATIVE_RERANKER_POSITIVE_TOKEN: Positive token (default: "yes").
      • GENERATIVE_RERANKER_NEGATIVE_TOKEN: Negative token (default: "no").

    2. Listwise Loss:

      • Uses Multi-class cross-entropy.
      • LISTWISE_RERANKER_TEMPERATURE: Listwise temperature (default: 1.0).
      • LISTWISE_RERANKER_MIN_GROUP_SIZE: Minimum group size (default: 2).

  3. Best practices for using instructions with Qwen3 Embedding

    main

    Both embedding and reranking models in the Qwen3 series are Instruction Aware. Using tailored instructions can improve performance by 1% to 5%.

    Guidelines for instructions:

    • Queries: Each query should be accompanied by a one-sentence instruction describing the task.
    • Retrieval Documents: There is no need to add instructions for the documents being retrieved.
    • Language: In multilingual contexts, it is recommended to write instructions in English, as most instructions used during training were in English.

  4. Use Qwen3 Reranker with vLLM

    main

    For high-performance reranking, use vllm>=0.8.5. This approach utilizes the LLM.generate method with SamplingParams to extract logprobs for the "yes" and "no" tokens.

    Key Configuration:

    • SamplingParams: Set temperature=0, max_tokens=1, and allowed_token_ids to only include the IDs for your true_token and false_token.
    • Logprob Extraction: The score is calculated by taking the exp of the logprobs for the true_token and false_token and normalizing them.
    • Chat Template: Use tokenizer.apply_chat_template to format the system and user messages correctly.
    from vllm import LLM, SamplingParams
from vllm.inputs.data import TokensPrompt

# ... (setup code for messages and suffix_tokens) ...

def compute_logits(model, messages, sampling_params, true_token, false_token):
    outputs = model.generate(messages, sampling_params, use_tqdm=False)
    scores = []
    for i in range(len(outputs)):
        final_logits = outputs[i].outputs[0].logprobs[-1]
        # ... (extract true_logit and false_logit from final_logits) ...
        true_score = math.exp(true_logit)
        false_score = math.exp(false_logit)
        score = true_score / (true_score + false_score)
        scores.append(score)
    return scores

# Sampling configuration
sampling_params = SamplingParams(
    temperature=0, 
    max_tokens=1,
    logprobs=20, 
    allowed_token_ids=[true_token, false_token],
)

  5. Install SWIFT for Qwen3-Embedding training

    main

    To train Qwen3-Embedding models, you must install the ModelScope SWIFT framework. You can install the stable version via pip or install from source for the latest features. It is also recommended to install transformers and several optional packages for performance optimization.

    Optional packages:

    • deepspeed: For multi-GPU training.
    • liger-kernel: To save GPU memory.
    • flash-attn: For acceleration (requires --no-build-isolation).
    # Install stable version
pip install ms-swift -U

# Install from source
pip install git+https://github.com/modelscope/ms-swift.git

pip install transformers -U

# Optional packages
pip install deepspeed
pip install liger-kernel
pip install flash-attn --no-build-isolation

  6. Evaluate Embedding Models using MTEB scripts

    main

    Run the run_mteb.sh script to evaluate embedding models against specific MTEB benchmarks.

    Arguments:

    • model_path: Path or name of the model weights (e.g., Qwen/Qwen3-Embedding-0.6B).
    • model_name: Name used for the result directory.
    • benchmark_name: The benchmark to use. Supported values are: MTEB(eng, v2), MTEB(cmn, v1), MTEB(Code, v1), and MTEB(Multilingual, v2).

    Output: Results are saved in results/${model_name}/${model_name}/no_revision_available. Each task's results are stored in individual JSON files.

    bash run_mteb.sh ${model_path} ${model_name} ${benchmark_name}

  7. Use Qwen3 Embedding with Sentence Transformers

    main

    To use the model via the sentence-transformers library, ensure transformers>=4.51.0 and sentence-transformers>=2.7.0 are installed.

    Usage Patterns:

    • Prompting: Queries benefit from using a prompt. You can use the built-in model.prompts["query"] or pass a custom prompt argument to model.encode().
    • Documents: Documents should be encoded without a specific prompt.
    • Optimization: For acceleration, use attn_implementation="flash_attention_2" and set padding_side="left" in tokenizer_kwargs.
    from sentence_transformers import SentenceTransformer

# Recommended configuration for acceleration
model = SentenceTransformer(
    "Qwen/Qwen3-Embedding-0.6B",
    model_kwargs={"attn_implementation": "flash_attention_2", "device_map": "auto"},
    tokenizer_kwargs={"padding_side": "left"},
)

queries = ["What is the capital of China?", "Explain gravity"]
documents = ["The capital of China is Beijing.", "Gravity is a force..."]

with torch.no_grad():
    # Use prompt_name="query" for queries
    query_embeddings = model.encode(queries, prompt_name="query")
    document_embeddings = model.encode(documents)
    
    similarity = model.similarity(query_embeddings, document_embeddings)

  8. Summarize Experimental Results

    main

    Use the summary.py script to aggregate and summarize results for embedding or reranking models.

    For Embedding Models:

    python3 summary.py results/${embedding_model_name}/${embedding_model_name}/no_version_available benchmark_name

    For Reranking Models:

    python3 summary.py results/${reranking_model_name}/no_model_name/no_version_available benchmark_name
    python3 summary.py results/${embedding_model_name}/${embedding_model_name}/no_version_available benchmark_name
python3 summary.py results/${reranking_model_name}/no_model_name/no_version_available benchmark_name

  9. Evaluate Reranking Models using MTEB scripts

    main

    Run the run_mteb_reranking.sh script to evaluate reranking models. Note that reranking evaluation depends on the recall results produced during the embedding evaluation phase.

    Arguments:

    • model_path: Path to the reranking model weights (e.g., Qwen/Qwen3-Reranker-0.6B).
    • model_name: Name used for the result directory.
    • retrieval_path: Path to the retrieval results generated during the embedding evaluation phase. This is typically results/${embedding_model_name}.
    • benchmark: The benchmark name.

    Output: Results are saved in results/${model_name}/no_model_name_available/no_revision_available/.

    bash run_mteb_reranking.sh ${model_path} ${model_name} ${retrieval_path} ${benchmark}

  10. Use Qwen3 Embedding with vLLM

    main

    For high-throughput embedding generation, use vllm>=0.8.5. Initialize the LLM class with task="embed".

    Workflow:

    1. Format queries with instructions using get_detailed_instruct.
    2. Pass the combined list of queries and documents to model.embed().
    3. Extract embeddings from the output objects via o.outputs.embedding.
    import torch
import vllm
from vllm import LLM

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery:{query}'

task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [get_detailed_instruct(task, 'What is the capital of China?'), get_detailed_instruct(task, 'Explain gravity')]
documents = ["The capital of China is Beijing.", "Gravity is a force..."]
input_texts = queries + documents

model = LLM(model="Qwen/Qwen3-Embedding-0.6B", task="embed")

outputs = model.embed(input_texts)
embeddings = torch.tensor([o.outputs.embedding for o in outputs])

  12. Prepare data for Reranker training

    main

    Reranker training data focuses on ranking relationships between query-document pairs. The format uses messages for the query, positive_messages for relevant documents, and negative_messages for irrelevant documents.

    Data Format:

    {"messages": [{"role": "user", "content": "query"}], "positive_messages": [[{"role": "assistant", "content": "relevant_doc1"}, {"role": "assistant", "content": "relevant_doc2"}]], "negative_messages": [[{"role": "assistant", "content": "irrelevant_doc1"}, {"role": "assistant", "content": "irrelevant_doc2"}]]}

    Important Memory Note: Each data item is expanded into MAX_POSITIVE_SAMPLES × (1 + MAX_NEGATIVE_SAMPLES) data points within the same batch. You must adjust per_device_train_batch_size downwards to avoid OOM errors based on these multipliers.