DOCSEARCHbyAlgolia
Submit Docs

Qwen3-Omni Documentation

repository·main·Indexed 26 days ago

https://github.com/qwenlm/qwen3-omni

A natively end-to-end multilingual omni-modal foundation model capable of processing text, image, audio, and video inputs to produce real-time streaming text and speech responses. Supports 19 languages for speech input and 10 for speech output. Includes specialized models such as Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner. Compatible with Hugging Face Transformers and vLLM for inference and deployment.

Tokens
13.8K
Snippets
29
Records
41
Agent score
85%

What's inside Qwen3-Omni

  1. Overview of Qwen3-Omni

    main
    Qwen3-Omni is a natively end-to-end multilingual omni-modal foundation model. It is designed to process diverse inputs including text, images, audio, and video, and provides real-time streaming responses in both text and natural speech.

  2. Overview of Qwen3-Omni capabilities

    main

    Qwen3-Omni is a natively end-to-end multilingual omni-modal foundation model capable of processing text, images, audio, and video. It provides real-time streaming responses in both text and natural speech.

    Key Capabilities:

    • Multimodal Support: Native support for text, image, audio, and video with high performance across benchmarks.
    • Multilingualism:
      • Speech Input: Supports 19 languages including English, Chinese, Korean, Japanese, German, Russian, Italian, French, Spanish, Portuguese, Malay, Dutch, Indonesian, Turkish, Vietnamese, Cantonese, Arabic, and Urdu.
      • Speech Output: Supports 10 languages including English, Chinese, French, German, Russian, Italian, Spanish, Portuguese, Japanese, and Korean.
    • Real-time Interaction: Low-latency streaming with natural turn-taking.
    • Flexible Control: Behavior can be customized via system prompts.
    • Specialized Models: Includes Qwen3-Omni-30B-A3B-Captioner for detailed, low-hallucination audio captioning.

  3. Manage the Qwen3-Omni Docker Container

    main

    If you are using the Docker-based deployment, use these commands to manage the container lifecycle:

    • Re-enter a container if you have exited it:
      docker start qwen3-omni
docker exec -it qwen3-omni bash
    • Remove the container completely:
      docker rm -f qwen3-omni
    docker start qwen3-omni
docker exec -it qwen3-omni bash

# Or to remove
docker rm -f qwen3-omni

  4. Run the Qwen3-Omni-30B-A3B-Captioner Web Demo

    main

    To launch the web demo for the Qwen3-Omni-30B-A3B-Captioner model using the Transformers backend and FlashAttention-2 support, run the web_demo_captioner.py script with the appropriate flags. Once running, the terminal will provide a local URL (e.g., http://127.0.0.1:8901/) which you can open in your browser to access the Web UI.

    python web_demo_captioner.py -c Qwen/Qwen3-Omni-30B-A3B-Captioner --use-transformers --flash-attn2

  5. Configure input sequence for multimodal evaluation

    main

    When evaluating Qwen3-Omni models with multimodal data (audio, image, video), the text prompt should be input as user text. Unless specified otherwise by a specific benchmark, the text content must follow the multimodal data in the sequence.

    Example input structure:

    messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "/path/to/audio.wav"},
            {"type": "image", "image": "/path/to/image.png"},
            {"type": "video", "video": "/path/to/video.mp4"},
            {"type": "text", "text": "Describe the audio, image and video."},
        ],
    },
]

  6. Run the Web Demo via Docker

    main

    To launch the web demo inside a Docker container, execute the web_demo.py script with the desired model checkpoint and server configuration.

    Use the following command:

    python web_demo.py -c Qwen/Qwen3-Omni-30B-A3B-Instruct --server-port 80 --server-name 0.0.0.0

  7. Best Practices for Qwen3-Omni-30B-A3B-Thinking

    main

    To maximize the reasoning capabilities of the Thinking model, always include an explicit textual instruction or task description in each dialogue round alongside multimodal inputs (audio, image, video).

    messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "/path/to/audio.wav"},
            {"type": "image", "image": "/path/to/image.png"},
            {"type": "video", "video": "/path/to/video.mp4"},
            {"type": "text", "text": "Analyze this audio, image, and video together."},
        ],
    }
]

  8. Launch Local Web UI Demo

    main

    You can run a web-based UI demo for Qwen3-Omni models.

    Prerequisites:

    • Install ffmpeg on your system.
    • Install Python dependencies: pip install gradio==5.44.1 gradio_client==1.12.1 soundfile==0.13.1.
    • For best performance, use the vLLM backend.

    Running the Demo: Use web_demo.py for Instruct and Thinking models, and web_demo_captioner.py for the Captioner model. Use the --use-transformers flag if you are not using vLLM.

    # For Qwen3-Omni-30B-A3B-Instruct with vLLM backend
python web_demo.py -c Qwen/Qwen3-Omni-30B-A3B-Instruct

# For Qwen3-Omni-30B-A3B-Instruct with Transformers backend
python web_demo.py -c Qwen/Qwen3-Omni-30B-A3B-Instruct --use-transformers --generate-audio

# For Qwen3-Omni-30B-A3B-Thinking with vLLM backend
python web_demo.py -c Qwen/Qwen3-Omni-30B-A3B-Thinking

# For Qwen3-Omni-30B-A3B-Captioner with vLLM backend
python web_demo_captioner.py -c Qwen/Qwen3-Omni-30B-A3B-Captioner

  9. Serve Qwen3-Omni using vLLM

    main

    You can deploy Qwen3-Omni models using the vllm serve command. Use the -tp flag to specify the number of GPUs (tensor parallelism) and --max-model-len to set the maximum sequence length. For multimodal capabilities, ensure --allowed-local-media-path / is set to allow access to local media files.

    # Qwen3-Omni-30B-A3B-Instruct for single GPU
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path / -tp 1

# Qwen3-Omni-30B-A3B-Instruct for multi-GPU (example on 4 GPUs)
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 65536 --allowed-local-media-path / -tp 4

# Qwen/Qwen3-Omni-30B-A3B-Thinking for single GPU
vllm serve Qwen/Qwen3-Omni-30B-A3B-Thinking --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path / -tp 1

# Qwen/Qwen3-Omni-30B-A3B-Thinking for multi-GPU (example on 4 GPUs)
vllm serve Qwen/Qwen3-Omni-30B-A3B-Thinking --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 65536 --allowed-local-media-path / -tp 4

  10. Install Qwen3-Omni for Hugging Face Transformers

    main
    To use Qwen3-Omni with Transformers, install transformers (version 5.2.0 or later recommended), accelerate, and qwen-omni-utils. Ensure ffmpeg is installed on your system. For reduced GPU memory usage, install flash-attn (requires compatible hardware and loading in torch.float16 or torch.bfloat16).

  11. Deploy Qwen3-Omni using Docker

    main

    You can deploy Qwen3-Omni using pre-built Docker images from qwenllm/qwen3-omni. Ensure you have the NVIDIA Container Toolkit installed to allow Docker to access the GPU.

    When running the container, you must map your local workspace (containing model files and data) to the container's internal path /data/shared/Qwen3-Omni. The example below maps host port 8901 to container port 80.

    LOCAL_WORKDIR=/path/to/your/workspace
HOST_PORT=8901
CONTAINER_PORT=80
docker run --gpus all --name qwen3-omni \
    -v /var/run/docker.sock:/var/run/docker.sock -p $HOST_PORT:$CONTAINER_PORT \
    --mount type=bind,source=$LOCAL_WORKDIR,target=/data/shared/Qwen3-Omni \
    --shm-size=4gb \
    -it qwenllm/qwen3-omni:3-cu124

  12. Download Qwen3-Omni model weights

    main

    You can download model weights automatically during loading in Hugging Face Transformers or vLLM. For manual downloads, use ModelScope (recommended for users in Mainland China) or the Hugging Face CLI.

    # Download through ModelScope
pip install -U modelscope
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Instruct --local_dir ./Qwen3-Omni-30B-A3B-Instruct
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Thinking --local_dir ./Qwen3-Omni-30B-A3B-Thinking
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Captioner --local_dir ./Qwen3-Omni-30B-A3B-Captioner

# Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir ./Qwen3-Omni-30B-A3B-Instruct
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Thinking --local-dir ./Qwen3-Omni-30B-A3B-Thinking
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Captioner --local-dir ./Qwen3-Omni-30B-A3B-Captioner