DOCSEARCHbyAlgolia
Submit Docs

HailoRT

repository·master·Indexed 41 days ago

https://github.com/hailo-ai/hailort

A production-grade runtime library version 5.4.0 designed to interface host processors with Hailo AI Inference Accelerators and Vision Processors. It provides C/C++, Python, CLI, and GStreamer interfaces to manage hardware, run inferences, and collect device telemetry. The library includes a GenAI API for Large Language Models (LLM), Vision-Language Models (VLM), and Speech2Text applications.

Tokens
18.5K
Snippets
64
Records
72
Agent score
53%

What's inside HailoRT

  1. Overview of HailoRT components

    master

    HailoRT is a lightweight, production-grade runtime library for host processors that provides optimized performance for Hailo AI hardware. It consists of several key components:

    • HailoRT Library: A user-space library providing intuitive C/C++ APIs.
    • HailoRT CLI: A command-line application for controlling Hailo devices, running inferences, and collecting statistics or device events.
    • pyHailoRT: A Python API that wraps the core runtime library.
    • HailoRT PCIe Driver: The device driver used for device management, communication, and data transfer (includes Hailo-8 firmware).
    • HailoRT GStreamer element (HailoNet): Integration for GStreamer-based media pipelines.

  2. Format multi-modal VLM structured prompts

    master

    VLM structured prompts use a JSON-based message format to support both text and image content within a single conversation. Unlike LLMs which use a single string, VLMs use an array of content items. Each message must specify a role (user, system, or assistant) and a content array containing objects with a type.

    {
    "role": "user|system|assistant",
    "content": [
        {
            "type": "text",
            "text": "text content"
        },
        {
            "type": "image"
        }
    ]
}

  3. Validate image count for VLM messages

    master

    The VLM API performs automatic validation to ensure the number of image frames provided matches the number of image type entries in the structured prompt. If you include an image type in your JSON content but do not provide a corresponding hailort::MemoryView in the frames vector, the generation will fail.

    // Valid: 2 images in message, 2 frames provided
std::vector<std::string> messages = {
    R"("{\"role\": \"user\", \"content\": [{"type": "image"}, {"type": "image"}, {"type": "text", "text": "Compare these"}]})"
};
std::vector<hailort::MemoryView> frames = {img1, img2};

// Invalid: 1 image in message, 0 frames provided - will cause error
std::vector<std::string> messages = {
    R"("{\"role\": \"user\", \"content\": [{"type": "image"}, {"type": "text", "text": "Compare these"}]})"
};
std::vector<hailort::MemoryView> frames = {};

  4. Manage Conversation Context and History

    master

    When building interactive chat applications, you must manage the conversation state to ensure the model has access to previous turns or can start fresh.

    Maintaining History

    To provide the model with context of a multi-turn conversation, include previous user and assistant messages in the vector passed to generator.write().

    Clearing Context

    To prevent memory issues during long sessions or to start a completely new topic without previous influence, use the clear_context() method.

    // Clear conversation history to free memory or reset state
llm.clear_context();
    // Multi-turn conversation example
std::vector<std::string> conversation = {
    R"("{\"role\": \"system\", \"content\": \"You are a helpful programming assistant\"})",
    R"("{\"role\": \"user\", \"content\": \"How do I sort an array in Python?\"})",
    R"("{\"role\": \"assistant\", \"content\": \"You can use the sorted() function or .sort() method\"})",
    R"("{\"role\": \"user\", \"content\": \"What's the difference between them?\"})
```;

generator.write(conversation);

// Later, to reset:
llm.clear_context();

  5. Compile and run the LLM Chat Raw Prompts example

    master

    This example demonstrates how to build an interactive chat application using Hailo's LLM API with raw (unstructured) prompts. This approach provides full control over prompt formatting and conversation structure, which is useful for advanced users needing fine-grained control over model behavior.

    Prerequisites

    • HailoRT installed and configured.
    • An appropriate LLM HEF file from Hailo Model Zoo GenAI.
    • Knowledge of the specific prompt template required by your model (e.g., Qwen2.5-1.5B-Instruct).

    Compilation

    You can compile the example from either the main examples directory or the specific example directory.

    From the examples directory:

    cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target chat_raw_prompt_example

    From the chat_raw_prompt_example directory:

    cd genai/chat_raw_prompt_example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

    Execution

    Run the compiled binary:

    ./build/cpp/chat_raw_prompt_example/chat_raw_prompt_example
    # From examples directory
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target chat_raw_prompt_example

# Or from chat_raw_prompt_example directory
cd genai/chat_raw_prompt_example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

# Execution
./build/cpp/chat_raw_prompt_example/chat_raw_prompt_example

  6. Run the Speech2Text GenAI example

    master

    The Speech2Text example demonstrates using the HailoRT GenAI C++ API to transcribe audio files using a Whisper-based model on Hailo hardware. The application loads a Speech2Text HEF model, accepts an audio file path, and outputs transcribed text segments with timestamps.

    Prerequisites

    • HailoRT: Must be installed and configured.
    • Model: An appropriate Speech2Text HEF file obtained from the Hailo Model Zoo GenAI.
    • Audio Format: The input audio file must strictly follow these specifications:
      • Format: PCM float32
      • Normalization: [-1.0, 1.0)
      • Channels: Mono
      • Endianness: Little-endian
      • Sample Rate: 16 kHz
    # Example execution command
./build/cpp/speech2text_example/speech2text_example <hef_path> <input_audio_file_path>

  7. Select the correct HailoRT branch for your device

    master

    The available branches in the repository are hardware-specific. Ensure you use the branch that matches your device family:

    • master branch: Supports only the Hailo-10 and Hailo-15 device families.
    • hailo8 branch: Use this branch for Hailo-8, Hailo-8R, and Hailo-8L devices.

  8. Compile HailoRT examples with CMake

    master

    To compile the HailoRT examples, use CMake. It is highly recommended to compile libhailort from source simultaneously to ensure include path and ABI compatibility.

    Note: You must have write permissions in the examples directory to compile. If you do not, copy the directory to a location where you have write access before proceeding.

    # Configure and build all examples
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

# Build a specific C++ example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target cpp_async_infer_basic_example

  9. Compile and run the VLM example

    master

    To use the Vision-Language Model (VLM) example, you must have HailoRT installed and a VLM HEF file from the Hailo Model Zoo GenAI. You can compile the example from either the main examples directory or the specific vlm_example directory using CMake.

    # From examples directory
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target vlm_example

# Or from vlm_example directory
cd genai/vlm_example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

# Execution
./build/cpp/vlm_example/vlm_example <hef_path>

  10. Compile the Speech2Text example

    master

    You can compile the Speech2Text example using CMake from either the main examples directory or the specific example directory.

    # Option 1: From the examples directory
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target speech2text_example

# Option 2: From the speech2text_example directory
cd genai/speech2text_example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

  11. Use Structured Prompts for LLM Conversations

    master

    Instead of manually constructing plain text strings with special tokens (e.g., <start>system:...<end>), it is recommended to use Structured Prompts. This approach uses a JSON-based message format that allows the HailoRT GenAI API to handle model-specific template application automatically.

    JSON Message Format

    Each message in the conversation must be a JSON string with the following structure:

    {
    "role": "user|system|assistant",
    "content": "message content"
}

    Roles:

    • system: Used for setting instructions and global context.
    • user: Used for user inputs and queries.
    • assistant: Used to represent previous model responses to maintain conversation history.

    Implementation Pattern

    Use generator.write() to pass a vector of JSON-formatted strings, then call generator.generate() to receive the completion.

    std::vector<std::string> messages = {
    R"("{\"role\": \"system\", \"content\": \"You are a helpful assistant\"})",
    R"("{\"role\": \"user\", \"content\": \"Hello\"})
```;

generator.write(messages);
auto completion = generator.generate();

  12. Run the LLM Chat Example

    master

    The LLM Chat Example demonstrates interactive conversation with Large Language Models using Hailo's GenAI API. It supports structured prompts via JSON to manage conversation history and roles effectively.

    Prerequisites

    • HailoRT installed and configured.
    • An appropriate LLM HEF file obtained from the Hailo Model Zoo GenAI.

    Compilation

    You can compile the example from either the root examples directory or the specific chat_example directory.

    From the examples directory:

    cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release --target chat_example

    From the chat_example directory:

    cd genai/chat_example
cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build --config release

    Execution

    Run the compiled binary by providing the path to your LLM HEF file:

    ./build/chat_example <hef_path>
    #!/bin/bash
./build/chat_example <hef_path>