OpenVINO GenAI Documentation

The Who What Benchmark (WWB) is a simple accuracy benchmarking tool for Generative AI models. Its primary purpose is to estimate the similarity score between embeddings computed from data generated by two different models (e.g., a baseline model vs. an optimized/quantized model).

Supported Use Cases

• LLMs (Text Generation): Supports HuggingFace Transformers (including Bitsandbytes, GPTQ), OpenVINO/NNCF via Optimum-Intel, OpenVINO GenAI, and Llama.cpp via BigDL-LLM.
• Text-to-Image Pipelines: Validates pipelines by computing similarity scores between images generated by Diffusers, Optimum-Intel, and OpenVINO GenAI using the Text2ImageEvaluator class.
• Visual Language Pipelines (VLMs): Validates pipelines using the VisualTextEvaluator class.
• Text-to-Video: Evaluates video generation (defaults guidance_scale to 1; higher values enable CFG).

OpenVINO™ GenAI is a library of optimized generative AI model pipelines, execution methods, and samples built on top of the OpenVINO Runtime. It provides simplified APIs that abstract the complexities of the generation process (such as tokenization, scheduling, and generation loops) to allow for easy integration of state-of-the-art models with minimal code.

Key characteristics include:

• Lightweight: Designed for efficient inference on PCs and laptops with a minimal memory footprint and no external dependencies (it includes core functionality like tokenization via openvino-tokenizers).
• Optimized: Hardware-specific optimizations for CPU, GPU, and NPU devices.
• Multi-language: Supports Python, C++, and Node.js.

The Misaki C++ port includes small API examples demonstrating core functionality:

• basic_usage.cpp: Demonstrates creating an English engine, calling phonemize_with_tokens(...), and printing phoneme output along with token fields.
• fallback_and_unknown.cpp: Demonstrates handling unknown words by using set_unknown_token(...) and installing a set_fallback_hook(...) to resolve fallback logic.

The package provides pre-built pipelines for several AI scenarios:

• Text generation (LLMs): Chat with models like Llama, Phi, Qwen, etc.
• Visual processing (VLMs): Analyze images/videos with LLaVa, MiniCPM-V, etc.
• Image generation (Diffusers): Generate images with Stable Diffusion & Flux.
• Speech recognition (Whisper): Convert speech to text.
• Speech generation (SpeechT5): Convert text to speech.
• Semantic search (Text Embedding): Compute embeddings for RAG workflows.
• Text Rerank (RAG): Analyze relevance of documents and queries for RAG.

LoRA (Low-Rank Adaptation) is a lightweight fine-tuning technique for Large Language Models (LLMs) and Stable Diffusion models. Instead of retraining all parameters of a massive model, LoRA adds a small number of new weights using low-rank matrix factorization.

Key benefits for developers:

• Efficiency: Drastically reduces the number of trainable parameters, making fine-tuning faster and more memory-efficient.
• Portability: Produces small weight files (typically a few hundred MBs) that are easy to store and share.
• Task Switching: Since the original pre-trained weights remain frozen, you can maintain multiple lightweight LoRA adapters for different tasks and switch between them easily.
• No Inference Latency: LoRA weights can be merged with the base model, ensuring that using an adapter does not slow down model inference.

LoRA (Low-Rank Adaptation) support varies depending on the pipeline type:

• LLMs: Supported.
• VLMs: Supported only for LoRA adapters applied to the language-model (LLM) part. Adapters targeting the vision encoder or other multimodal components are not supported.
• Speech Recognition: Not supported.
• Speech Generation: Not supported.
• Embedding Models: Not supported.
• Text Rerank Models: Not supported.

When enable_prefix_caching=true is used in hybrid models, the cache_interval_multiplier controls the trade-off between memory usage and prefix-cache reuse granularity.

• Smaller multiplier: Creates more frequent checkpoints. This results in finer-grained reuse (better for chat/repeated prompts) but consumes more linear-attention memory.
• Larger multiplier: Creates fewer checkpoints. This results in coarser reuse but reduces linear-attention memory usage.

If unset, the runtime derives the multiplier adaptively so that one checkpoint costs roughly one KV block (with a minimum baseline of 8).

OpenVINO GenAI provides the VLMPipeline for multimodal text-generation Vision Language Models (VLMs). This pipeline allows you to generate text responses based on a text prompt combined with image and/or video inputs. The pipeline is available for Python, C++, and JavaScript (Node.js) implementations and supports execution on both CPU and GPU devices.

# Conceptual usage of VLMPipeline
# The pipeline handles multimodal inputs (text + images/videos)
from openvino_genai import VLMPipeline

pipeline = VLMPipeline(model_path, device="CPU")
result = pipeline.generate("prompt", image=image_data)

OpenVINO GenAI is positioned as a middle ground between the low-level OpenVINO Runtime and high-level frameworks like Hugging Face Optimum Intel:

Key Takeaways:

• Vs. Base OpenVINO: OpenVINO GenAI provides ready-to-use components (generation loops, tokenization, scheduling) that must otherwise be manually implemented in base OpenVINO.
• Vs. Hugging Face Optimum Intel: OpenVINO GenAI offers a smaller footprint, fewer dependencies, and better optimization options, particularly for C++ applications.