OpenVINO Notebooks

This notebook provides a workflow for running Tencent's Hunyuan machine translation models using OpenVINO. It supports two model series:

1. HY-MT1.5: Based on the Hunyuan Dense V1 architecture.
   • HY-MT1.5-1.8B: Lightweight model for resource-constrained environments.
   • HY-MT1.5-7B: Larger model for higher translation quality.
2. Hy-MT2: Next-generation series with improved quality and coverage.
   • Hy-MT2-1.8B: Lightweight next-gen model.
   • Hy-MT2-7B: Higher quality next-gen model.

Note: This is an EXPERIMENTAL NOTEBOOK. The models may not have been fully validated with OpenVINO.

This tutorial demonstrates how to convert and optimize the Flex.2 model using OpenVINO. Flex.2 is a flexible text-to-image diffusion model based on the Flux architecture that supports inpainting and universal control (accepting pose, line, and depth inputs).

The workflow covered in the notebook includes:

1. Installing prerequisites.
2. Collecting the PyTorch model pipeline.
3. Converting the model to OpenVINO Intermediate Representation (IR) format.
4. Compressing weights using NNCF.
5. Preparing the OpenVINO Inference pipeline.
6. Running image generation.
7. Launching an interactive demo.

This notebook demonstrates how to use Qwen2-VL, a multimodal large language model, as a visual-language assistant. It covers converting and optimizing the model for a multimodal chatbot using Optimum Intel and applying weight compression techniques via NNCF.

Key capabilities of Qwen2-VL include:

• State-of-the-art understanding of images with various resolutions and ratios.
• Ability to understand videos longer than 20 minutes.
• Multilingual support for text within images (English, Chinese, European languages, Japanese, Korean, Arabic, Vietnamese, etc.).
• Complex reasoning for agentic tasks (e.g., operating mobile phones or robots).

This tutorial demonstrates how to run an instruction-following text generation pipeline using Large Language Models (LLMs) optimized with OpenVINO.

Key components used:

• Hugging Face Transformers: For pre-trained models.
• Hugging Face Optimum Intel: To convert models to OpenVINO IR format.
• OpenVINO NNCF: To compress model weights to INT8 and INT4.
• OpenVINO Generate API: To create and run the instruction-following inference pipeline.

Supported models include:

• tiny-llama-1b-chat (1.1B parameters)
• phi-2 (2.7B parameters)
• red-pajama-3b-instruct (2.8B parameters)
• mistral-7b (7B parameters)

GLM-OCR is a 0.9B-parameter vision-language model by Zhipu AI designed for document parsing. It specializes in three specific tasks:

• Text Recognition
• Formula Recognition
• Table Recognition

This notebook demonstrates how to convert GLM-OCR to OpenVINO Intermediate Representation (IR), compress it to 4-bit weights using NNCF, and run inference on Intel CPU, integrated GPU, or Arc GPU using optimum-intel. It also includes a Gradio demo and an optional PP-DocLayout-V3 pipeline for document region detection.

This tutorial demonstrates how to run the Phi-4-multimodal-instruct model using OpenVINO and optimize it using NNCF.

Phi-4-multimodal-instruct is a 5.6B parameter multimodal transformer model that processes text, image, and audio inputs to generate text outputs. It uses the pretrained Phi-4-mini as its backbone language model, integrated with vision and speech encoders and adapters.

The notebook workflow includes:

1. Installing requirements.
2. Converting and optimizing the model.
3. Running OpenVINO model inference.
4. Launching an interactive chatbot demo that answers questions about provided images and audio content.

HunyuanOCR is a lightweight, end-to-end vision-language model (VLM) designed for document parsing, text spotting, information extraction, and text-image translation.

This notebook demonstrates how to:

• Convert HunyuanOCR to OpenVINO Intermediate Representation (IR) using optimum-intel.
• Optionally compress weights to INT8 (or keep FP16) using NNCF.
• Run inference on Intel CPU, integrated GPU, or Arc GPU.
• Use a streaming Gradio demo with task presets aligned with the official HunyuanOCR Space.

Gemma 4 is a family of open multimodal models from Google DeepMind that supports text and image input with text output. The family includes different model sizes and architectures:

• E2B: 5.1B Total Params (2.3B Active), Text/Image/Audio, Dense + PLE architecture.
• E4B: 8B Total Params (4.5B Active), Text/Image/Audio, Dense + PLE architecture.
• 26B-A4B: 25.2B Total Params (3.8B Active), Text/Image, MoE architecture.
• 31B: 30.7B Total Params (30.7B Active), Text/Image, Dense architecture.

Key features include built-in chain-of-thought (thinking mode), native system prompt support, and interleaved multi-image input.

FLUX.2 [klein] is a 4-billion-parameter rectified flow transformer from Black Forest Labs that unifies text-to-image generation and image editing. This notebook demonstrates how to convert and optimize the model using OpenVINO with INT4 weight compression.

Key Capabilities:

• Text-to-image generation and multi-reference image editing in a single model.
• High performance: Sub-second generation using distilled 4-step mode.
• Hardware requirements: Runs on consumer GPUs with approximately 13 GB VRAM.
• License: Open weights under Apache 2.0.

This tutorial demonstrates how to convert and optimize the Qwen2.5-Omni multimodal model for use with OpenVINO. Qwen2.5-Omni is an end-to-end model capable of perceiving text, images, audio, and video, and generating text and natural speech responses in a streaming manner.

The tutorial covers the following workflow:

1. Installing requirements.
2. Downloading the PyTorch model.
3. Converting the model to OpenVINO Intermediate Representation (IR).
4. Compressing Language Model weights using NNCF.
5. Running OpenVINO model inference.
6. Launching an interactive demo (e.g., a chatbot that answers questions about image content).