OpenVINO Model Server (OVMS)

OpenVINO™ Model Server (OVMS) is a high-performance C++ system designed for serving models via standard network protocols (REST and gRPC). It enables remote inference, allowing lightweight clients to perform API calls to edge or cloud deployments without needing the model framework or hardware-specific drivers locally.

Key architectural benefits include:

• Framework Independence: Client applications are decoupled from the model framework and hardware device.
• Security: Model topology and weights are not exposed directly to clients.
• Scalability: Supports horizontal and vertical inference scaling, making it ideal for microservices in Kubernetes or OpenShift.
• Interoperability: Supports multiple frameworks (TensorFlow, PaddlePaddle, ONNX) and provides APIs compatible with OpenAI, Cohere, KServe, and TensorFlow Serving.

OpenVINO Model Server (OVMS) provides a /v3 frontend specifically designed for Generative AI use cases. These endpoints are compatible with OpenAI and Cohere API standards and can be extended using MediaPipe graphs.

OpenAI Compatible Endpoints

• chat/completions
• completions
• embeddings
• images/generations
• images/edit
• audio/transcriptions
• audio/translations
• audio/speech
• tokenize
• /models
• /models/{model}

Cohere Compatible Endpoints

• rerank

OVMS supports complex inference workflows using MediaPipe graphs and Directed Acyclic Graph (DAG) pipelines:

• MediaPipe Graphs: Implement chains of models for tasks like object detection, iris detection, and holistic tracking.
• DAG Pipelines: Use the pipeline feature with custom nodes and demultiplexers for advanced workflows:
  • OCR: Horizontal text detection and full Optical Character Recognition (OCR) pipelines.
  • Face Analysis: Single or multi-face analysis (age, gender, emotion) and face blurring.
  • Vehicle Analysis: Detection and attribute recognition.
  • Ensembles: Combining multiple classification models to improve accuracy.

OpenVINO Model Server (OVMS) provides efficient serving for generative AI workloads by integrating state-of-the-art optimization techniques from the OpenVINO GenAI Library. These optimizations include:

• Continuous Batching
• Paged Attention
• Dynamic Split Fuse

For a basic implementation example, refer to the quickstart guide.

OpenVINO Model Server (OVMS) provides several specialized demos for Generative AI workloads, including LLMs, VLMs, and Image Generation. Key capabilities include:

• LLM & VLM Serving: Support for continuous batching pipelines to optimize text and vision-language model generation.
• API Compatibility: Endpoints compatible with OpenAI API for text embeddings and Cohere API for reranking.
• Hardware Acceleration: Specific guidance for running LLMs and VLMs on NPUs.
• Integration: Support for Open WebUI as an inference provider and Visual Studio Code (via the Continue extension) for local AI assistance.
• Advanced LLM Features: Support for GGUF models and recommendations for handling long-context LLMs.

OpenVINO Model Server (OVMS) can serve speech generation and speech recognition models using OpenAI-compatible endpoints. Supported endpoints include:

• audio/speech (Speech generation)
• audio/transcriptions (Speech-to-text)
• audio/translations (Speech-to-text translation)

Note that audio/transcriptions supports speech-to-text streaming responses. This demo specifically focuses on the Kokoro model for speech generation.

This demo provides a workflow to integrate Open WebUI with OpenVINO Model Server (OVMS). Open WebUI serves as the user interface for generative models, while OVMS handles the execution of models (text generation, embeddings, and reranking) via standard APIs compatible with OpenAI.

Prerequisites

• Architecture: x86_64
• OS: Linux (requires Docker Engine) or Windows
• Python: 3.11 with pip
• Account: HuggingFace account for model downloads

Deployment Options

• Linux: Use Docker containers.
• Windows: Use the binary package.
• Hardware Note: For hosts with limited VRAM, consider using a subset of models or changing the target_device to CPU or NPU.

OpenVINO Model Server (OVMS) can be deployed using several different methods depending on your infrastructure requirements:

• Docker: Use pre-built container images from Docker Hub or the Red Hat Ecosystem Catalog, or build a custom image from source.
• Baremetal (Linux or Windows): Download the packaged binary and run it directly on your host system.
• Kubernetes: Deploy using a Helm chart, the Kubernetes Operator, or the OpenShift Operator.

Once deployed, the server can be configured via various startup modes, parameters, and accelerator settings.

OpenVINO™ Model Server provides several API interfaces to allow integration with different inference workflows. Depending on your use case, you can use one of the following:

• KServe API: Used for general inference. It supports both gRPC and REST interfaces.
• OpenAI API: Used for Generative AI tasks, specifically:
  • Text generation (Chat Completions and Completions)
  • Embeddings
  • Image generation
  • Speech recognition (Speech-to-Text)
  • Speech generation (Text-to-Speech)
• Cohere API: Used for reranking tasks.

For REST-based generative use cases, the endpoints support both streamed and unary responses.