Edge AI Libraries

The Alert Agent Service is a multimodal alert action dispatcher microservice designed for edge environments. It acts as a bridge between detection pipelines (video analytics, audio sensors, IoT) and response actions.

How it works:

1. Receives Alerts: Accepts multimodal payloads including text, image, audio, video, and binary media.
2. Reasons: Uses an LLM-powered agentic backend (via Google ADK and OpenVINO™ Model Server) to analyze the alert context.
3. Dispatches Actions: Selects and executes configurable tools such as logging, webhooks, MQTT publishing, or snapshot saving.

Key Capabilities:

• Agentic vs. Rule-Based: Supports intelligent tool selection via LLM or deterministic routing via rules (set by AGENT_MODE=false).
• Deduplication: Prevents notification floods by suppressing repeated alerts within a configurable window.
• Escalation: Automatically triggers additional tools (e.g., MQTT) after a specific number of consecutive detections.
• Extensibility: Supports Model Context Protocol (MCP) to dynamically add tools from external servers and allows hot-reloading of configurations without restarts.

The KVCache Quantization and Offload Library provides a near-lossless 4-bit KV-cache quantization codec designed for LMCache and vLLM. It is specifically optimized for edge devices to offload KV caches from XPU memory to host memory or disk. This process restores prefix-cache hits that would otherwise be lost due to memory pressure, effectively turning full prefills into cache hits served from disk.

Key Features:

• 4-bit Quantization: Uses Randomized Hadamard Transform (RHT) preconditioning for near-lossless compression.
• Configurable Scaling: Supports per_tensor, per_channel, and per_token scaling methods, with optional asymmetric quantization.
• Multi-Platform Kernels: Includes CPU kernels (AVX2/AVX-512 with optional OpenMP) and optional standalone SYCL/DPC++ kernels for Intel XPUs.
• LMCache Integration: Acts as a drop-in codec for LMCache's serde interface, remaining transparent to the vLLM serving path.
• Hybrid Model Support: Allows per-sub-state control, enabling hybrid models (like Mamba or linear-attention) to keep specific states at full precision.

The Agent Quality Handler is a standalone, configuration-driven microservice designed to run Agentic Predictive Maintenance graphs. It processes detections received from an external storage API to perform automated analysis and decision-making.

Key Capabilities

• Agent Types: Supports Policy, Analysis, Evidence, and Ticketing agents.
• Execution Modes: Offers a rule-based fallback mode or a profile-gated LLM mode.
• Ingestion: Ingests detections via MQTT, supporting authenticated connections to external brokers.
• Extensibility: Supports custom use-case configurations and prompt mounts.
• Observability: Provides a direct REST API and Prometheus metrics on port 5002.
• Error Handling: Provides explicit graph errors while preserving partial results.

Deployment Notes

• The service requires an external Detection Service and storage.
• A default Docker Compose deployment includes the agent and a private MQTT broker.
• Activating the llm profile adds OpenVINO Model Server (OVMS) and model download capabilities.
• Note that the default deployment does not include a UI or Nginx.

ViPPET (Visual Pipeline and Platform Evaluation Tool) is a tool designed to simplify hardware selection for AI workloads. It allows developers to configure workload parameters, perform performance benchmarking, and analyze critical metrics to optimize hardware selection and performance tuning.

Core Capabilities

• Hardware Comparison: Compare how AI pipelines behave across different hardware targets, model variants, and input sources.
• Performance Benchmarking: Validate model performance targets and KPIs by testing inference pipelines with various accelerators to measure throughput, latency, and resource utilization.
• End-to-End Validation: Exercise complete media and AI pipelines with different inputs and execution settings to understand real-world deployment behavior.
• Intel® Optimization: Optimized for Intel® AI Edge Systems, allowing pipelines to run directly on target devices for seamless integration.

The Dataprep - Milvus microservice is a Python-based service designed to preprocess visual data (images and videos) and store their feature embeddings in a vector database (Milvus) for efficient retrieval.

It uses the CLIP model's image encoder to extract embeddings. This allows for advanced semantic search capabilities across large-scale visual datasets.

Core Capabilities:

• Video Processing: Automatically extracts frames at configurable intervals for embedding generation.
• Image/Frame Processing: Handles resizing, color conversion, and normalization.
• Object-Aware Preprocessing: Uses object detection to crop images around key subjects. This ensures that important objects remain recognizable and high-quality even after resizing, which significantly improves retrieval performance for complex scenes.
• Metadata Linking: Maintains a link between cropped objects and their original source images. If a search retrieves a specific cropped object, the system can return the original image to provide full context.