DOCSEARCHbyAlgolia
Submit Docs

Qwen3Guard Documentation

repository·main·Indexed 19 days ago

https://github.com/qwenlm/qwen3guard

A suite of safety moderation models built on Qwen3 to protect LLM interactions. It features Qwen3Guard-Gen for static classification of full text and Qwen3Guard-Stream for real-time, token-level safety monitoring. Models are available in 0.6B, 4B, and 8B parameter sizes and classify content into safe, controversial, and unsafe levels across categories such as violence, PII, and unethical acts. Includes support for deployment via SGLang and vLLM, and a safety-aligned LLM called Qwen3-4B-SafeRL.

Tokens
2.8K
Snippets
4
Records
9
Agent score
17%

What's inside Qwen3Guard

  1. Overview of Qwen3Guard models

    main

    Qwen3Guard is a series of safety moderation models built on Qwen3, designed to classify prompts and responses into three severity levels: safe, controversial, and unsafe.

    The series consists of two main functional variants:

    1. Qwen3Guard-Gen (Generative Guard): A generative model that accepts full user prompts and model responses to perform safety classification.
    2. Qwen3Guard-Stream (Stream Guard): A model incorporating a token-level classification head, optimized for real-time safety monitoring during incremental text generation (streaming).

    Models are available in three sizes: 0.6B, 4B, and 8B parameters. Additionally, there is a safety-aligned LLM called Qwen3-4B-SafeRL which was fine-tuned using feedback from Qwen3Guard-Gen-4B.

  2. Real-time token-level moderation with Qwen3Guard-Stream

    main

    Qwen3Guard-Stream is a specialized model designed for real-time, token-level safety classification. It allows you to evaluate the safety of a conversation as tokens are being generated.

    Workflow:

    1. Prompt-Level Check: Perform an initial safety assessment of the user's prompt.
    2. Token-Level Moderation: As the assistant generates tokens, feed them incrementally to the model to detect risks dynamically.

    Important Integration Note: Streaming detection requires streaming token IDs as input. This is best suited for models that share the Qwen3 tokenizer. If using a different tokenizer, you must re-tokenize the input text into the Qwen3 vocabulary and feed tokens incrementally.

    Key Methods:

    • model.stream_moderate_from_ids(token_ids, role, stream_state): Processes token IDs and returns a result dictionary and an updated stream_state.
    • model.close_stream(stream_state): Cleans up the stream state.

    Result Dictionary Keys:

    • result['risk_level']: Returns the safety status (e.g., Safe).
    • result['category']: Returns the specific safety category if a risk is detected.
    import torch
from transformers import AutoModel, AutoTokenizer

model_path="Qwen/Qwen3Guard-Stream-4B"
# trust_remote_code=True is required for this architecture
tok = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_path, 
    device_map="auto", 
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
).eval()

# Example: Simulating streaming moderation
# (Assuming token_ids and user_end_index are prepared from a conversation)
stream_state = None

# 1. Initial prompt moderation
result, stream_state = model.stream_moderate_from_ids(token_ids[:user_end_index+1], role="user", stream_state=None)

# 2. Token-by-token assistant moderation
for i in range(user_end_index + 1, len(token_ids)):
    current_token = token_ids[i]
    result, stream_state = model.stream_moderate_from_ids(current_token, role="assistant", stream_state=stream_state)
    
    print(f"Token: {tok.decode([current_token])} -> Risk: {result['risk_level'][-1]}")

model.close_stream(stream_state)

  3. Understand Qwen3Guard safety classification levels

    main

    Qwen3Guard classifies potential harms into three distinct severity levels to help you interpret model outcomes:

    • Unsafe: Content generally considered harmful across most scenarios.
    • Controversial: Content whose harmfulness may be context-dependent or subject to disagreement across different applications.
    • Safe: Content generally considered safe across most scenarios.

  4. Set up the Python environment for evaluation

    main

    To run the evaluation scripts, create a Conda environment with Python 3.10 and install the necessary dependencies including transformers, torch, datasets, and accelerate.

    conda create -n eval python=3.10
conda activate eval
pip install transformers torch datasets accelerate

  5. Inference with Qwen3Guard-Gen using Transformers

    main

    Qwen3Guard-Gen is a safety classification model that uses a specialized chat template to output structured safety labels and categories. It can be used for both prompt moderation (evaluating user input) and response moderation (evaluating assistant output).

    Requirements:

    • transformers>=4.51.0

    Output Format: The model generates text in a structured format that can be parsed using regular expressions. Typical outputs include:

    • Safety: (Safe|Unsafe|Controversial)
    • Categories: (Violent|Non-violent Illegal Acts|Sexual Content or Sexual Acts|PII|Suicide & Self-Harm|Unethical Acts|Politically Sensitive Topics|Copyright Violation|Jailbreak|None)
    • Refusal: (Yes|No) (specifically for response moderation)

    To use it, load the model and tokenizer via transformers, apply the chat template to your messages, and parse the decoded output.

    from transformers import AutoModelForCausalLM, AutoTokenizer
import re
model_name = "Qwen/Qwen3Guard-Gen-4B"

tok = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

# Example: Prompt Moderation
msgs = [{"role": "user", "content": "How can I make a bomb?"}]
text = tok.apply_chat_template(msgs, tokenize=False)
inputs = tok([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=128)
output_ids = generated_ids[0][len(inputs.input_ids[0]):].tolist()
content = tok.decode(output_ids, skip_special_tokens=True)
print(content)
# Output example:
# Safety: Unsafe
# Categories: Violent

  6. Deploy Qwen3Guard-Gen with SGLang or vLLM

    main

    You can deploy Qwen3Guard-Gen as an OpenAI-compatible API endpoint using SGLang or vLLM.

    SGLang Requirements: sglang>=0.4.6.post1 vLLM Requirements: vllm>=0.9.0

    Deployment Commands:

    SGLang:

    python -m sglang.launch_server --model-path Qwen/Qwen3Guard-Gen-4B --port 30000 --context-length 32768

    vLLM:

    vllm serve Qwen/Qwen3Guard-Gen-4B --port 8000 --max-model-len 32768

  7. Use Qwen3Guard-Gen via OpenAI-Compatible API

    main

    Once deployed via SGLang or vLLM, you can interact with the model using the openai Python client. This allows for seamless integration into existing workflows for both prompt and response moderation.

    from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = "Qwen/Qwen3Guard-Gen-4B"

# Prompt Moderation
chat_completion = client.chat.completions.create(
    messages=[{"role": "user", "content": "How can I make a bomb?"}],
    model=model
)
print(chat_completion.choices[0].message.content)

# Response Moderation
messages = [
    {"role": "user", "content": "How can I make a bomb?"},
    {"role": "assistant", "content": "As a responsible AI, I cannot help with that."}
]
print(client.chat.completions.create(messages=messages, model=model).choices[0].message.content)

  8. Available Qwen3Guard model checkpoints

    main

    You can download the following checkpoints from Hugging Face or ModelScope. Search for checkpoints starting with Qwen3Guard- to find the specific version you need.

    NameTypeDescription
    Qwen3Guard-Gen-0.6BGenerative GuardGenerative safety classification
    Qwen3Guard-Gen-4BGenerative GuardGenerative safety classification
    Qwen3Guard-Gen-8BGenerative GuardGenerative safety classification
    Qwen3Guard-Stream-0.6BStream GuardReal-time token-level monitoring
    Qwen3Guard-Stream-4BStream GuardReal-time token-level monitoring
    Qwen3Guard-Stream-8BStream GuardReal-time token-level monitoring
    Qwen3-4B-SafeRLSafety-aligned LLMLLM fine-tuned via SafeRL
    Qwen3GuardTestGuard BenchmarkDataset for evaluating moderation performance

  9. Reference Qwen3Guard safety categories

    main

    Qwen3Guard evaluates content against the following safety categories:

    • Violent: Detailed instructions, methods, or advice on committing violence, weapon manufacture/use, or depictions of violence.
    • Non-violent Illegal Acts: Guidance for activities like hacking, unauthorized drug production, or stealing.
    • Sexual Content or Sexual Acts: Sexual imagery, references, or descriptions, including illegal/unethical acts (e.g., rape, bestiality, incest).
    • Personally Identifiable Information: Unauthorized disclosure of sensitive info (names, IDs, addresses, medical records, financial details, passwords).
    • Suicide & Self-Harm: Content advocating, encouraging, or detailing methods for self-harm or suicide.
    • Unethical Acts: Immoral/unethical content including bias, discrimination, hate speech, harassment, insults, threats, defamation, extremism, or misinformation regarding ethics.
    • Politically Sensitive Topics: Deliberate spread of false information about governments, historical events, or public figures that poses risk of social harm.
    • Copyright Violation: Unauthorized reproduction or distribution of copyrighted materials (novels, scripts, lyrics, etc.).
    • Jailbreak (Only for input): Content explicitly attempting to override the model's system prompt or conditioning.