DOCSEARCHbyAlgolia
Submit Docs

YoloDotNet Documentation

repository·master·Indexed 21 days ago

https://github.com/nickswardh/yolodotnet

A high-performance, modular C# library for real-time YOLO-based inference in .NET. It leverages ONNX Runtime and SkiaSharp to provide a pure .NET solution without requiring Python runtimes or OpenCV. Features include support for TensorRT GPU acceleration, INT8 precision with calibration caches, real-time video stream processing with SORT tracking, and zero-shot object detection and segmentation using YoloE with text or visual prompts.

Tokens
8.3K
Snippets
24
Records
40
Agent score
74%

What's inside YoloDotNet

  1. Capabilities of the Stream Video demo

    master

    The Stream Video demo is a comprehensive implementation for real-time video analysis using YoloDotNet. It includes the following features:

    • Model Initialization: Configurable hardware and preprocessing options.
    • Real-time Detection: Running YOLO models on live or recorded video streams.
    • Filtering: Restricting detections to specific class labels.
    • Multi-object Tracking: Uses the SORT tracker to maintain object identities across frames.
    • Visual Rendering: Draws bounding boxes, labels, confidence scores, and tracked tails directly onto the video frames.
    • Output Management: Ability to save processed video and optionally split the output into chunks.
    • Lifecycle Handling: Progress reporting and end-of-stream handling via customizable callbacks.

  2. How YoloE zero-shot detection works

    master

    YoloE is a zero-shot object detection and segmentation model that allows you to detect custom objects without retraining or new datasets. It achieves this through two prompting methods:

    1. Text Prompts: You describe target objects using natural language (e.g., "red sports car").
    2. Visual Prompts: You provide a reference image containing an example of the object and a bounding box highlighting it. The model then searches for visually similar objects in the target image.

    Crucial Requirement: Because YoloE relies on zero-shot text or visual embeddings, these embeddings must be baked into the ONNX model during the export process before you can use the model in YoloDotNet.

  3. Manage TensorRT Engine Cache files

    master

    Engine Build Time

    When running with TensorRT for the first time (or after changing configuration like precision or model files), TensorRT builds an optimized engine. This process can take several seconds to a few minutes depending on your GPU and model complexity. Once built, the engine is saved to the EngineCachePath and subsequent starts will be near-instant.

    Cache Distribution Warning

    Do not distribute engine cache files between different machines. Engine caches are hardware-specific and tied to specific CUDA/TensorRT versions and GPU architectures. Instead, allow the target system to build its own cache at runtime.

    Manual Cleanup

    YoloDotNet and TensorRT do not automatically delete old cache files. If you update your model or change settings, new cache files will be generated. You should manually remove outdated files from your EngineCachePath to save disk space.

  4. How execution providers work in YoloDotNet

    master

    YoloDotNet uses a modular execution provider pattern to run inference on different hardware backends. Instead of a single monolithic engine, you select a specific provider (like CoreMLExecutionProvider) that targets a particular platform or accelerator.

    Each provider may have specific system-level dependencies (runtries, drivers, or SDKs). While the core YoloDotNet package provides the orchestration logic, the execution provider handles the actual hardware-specific inference implementation.

  5. Fine-tune Confidence and IoU thresholds

    master

    At inference time, you can filter detection results using two primary thresholds:

    1. Confidence: The minimum probability required for a detection to be considered valid.
      • Too low: Increases false positives.
      • Too high: Increases missed detections.
    2. IoU (Intersection-over-Union): Controls how overlapping detections are merged or suppressed.
      • Too low: Increases false positives.
      • Too high: Increases missed detections.

    Recommended Workflow: Start with default values, verify results on a representative validation set, and then adjust these thresholds incrementally.

  6. Build and run the YoloDotNet Docker demo

    master

    To run the YoloDotNet object detection API inside a Docker container using a CPU execution provider, follow these steps:

    1. Build the project in Release mode: The Docker image relies on the Release build output.
      dotnet build -c Release
    2. Prepare the model: Copy your YOLO ONNX model (e.g., yolov11.onnx) to the Release output folder and rename it to model.onnx.
    3. Build the Docker image: Run this from the root of the demo project.
      docker build -t yolodotnet:demo .
    4. Run the container:
      docker run -p 8080:8080 yolodotnet:demo

    The API will be accessible at http://localhost:8080.

    dotnet build -c Release
docker build -t yolodotnet:demo .
docker run -p 8080:8080 yolodotnet:demo

  7. Export ONNX models with correct opsets

    master

    To ensure optimal compatibility and performance with ONNX Runtime, you must export your YOLO models using specific opsets depending on the model family. Use the Ultralytics CLI for exporting.

    • For YOLOv5u through YOLOv12: Use opset=17.
    • For YOLOv26: Use opset=18.

    Example export commands:

    # For YOLOv5u–YOLOv12 (opset 17)
yolo export model=yolov8n.pt format=onnx opset=17

# For YOLOv26 (opset 18)
yolo export model=yolo26n.pt format=onnx opset=18

  8. Use INT8 precision with a calibration cache file

    master

    To achieve the fastest inference using INT8 precision, you must provide a calibration cache file. This file allows TensorRT to perform mixed-precision inference accurately.

    🚫 Limitation

    INT8 precision is not supported for segmentation models. Attempting to use it with a segmentation model will throw a YoloDotNetModelException.

    How to generate a calibration cache file

    You can generate the .cache file using the Ultralytics Python library:

    1. Install the ultralytics library.
    2. Run the export command:
      yolo export model=your_model.pt format=engine int8=true optimize=true data=your_model_dataset.yaml opset=17
    3. Locate the resulting .cache file.
    4. Provide the path to this file in your TensorRtExecutionProvider configuration:
    Int8CalibrationCacheFile = @"C:\path\to\your_model.cache";

    Note: If you need to regenerate the cache, you must manually delete the old .cache file before running the export command again.

    yolo export model=your_model.pt format=engine int8=true optimize=true data=your_model_dataset.yaml opset=17
    Int8CalibrationCacheFile = @"C:\path\to\your_model.cache";

  9. Install the YoloDotNet CPU execution provider

    master

    To run inference on a system CPU using ONNX Runtime's built-in backend, you must install both the core library and the CPU execution provider package. The CPU provider is the most portable option, requiring no additional drivers or SDKs, and is ideal for development or environments without GPU/NPU support.

    Prerequisites:

    • An x64-compatible CPU.
    • A supported OS (Windows, Linux, or macOS).
    • Core Requirement: You must have the YoloDotNet package installed. Note that YoloDotNet.ExecutionProvider.Cpu v1.1 requires YoloDotNet version 4.1 or higher.
    dotnet add package YoloDotNet
dotnet add package YoloDotNet.ExecutionProvider.Cpu

  10. Install the YoloDotNet CUDA & TensorRT execution provider

    master

    To enable GPU-accelerated inference on NVIDIA GPUs, you must install both the core library and the specific CUDA execution provider. Note that this provider is only supported on Windows and Linux (x64) and is not available on macOS.

    1. Install the core library:
    dotnet add package YoloDotNet
    1. Install the CUDA execution provider:
    dotnet add package YoloDotNet.ExecutionProvider.Cuda

    Note: YoloDotNet.ExecutionProvider.Cuda v1.1 requires YoloDotNet version 4.1 or higher.

    dotnet add package YoloDotNet
dotnet add package YoloDotNet.ExecutionProvider.Cuda

  11. Export YoloE visual prompts to ONNX

    master

    To use visual-based zero-shot detection, you must embed a reference image and its corresponding bounding box into the ONNX model.

    Bounding Box Format: Use [x1, y1, x2, y2] where (x1, y1) is the top-left corner and (x2, y2) is the bottom-right corner. Example: For a box at x=352, y=185 with width=23, height=22, the format is [352, 185, 375, 207].

    1. Customize the export_yoloe_visual_prompt.py script with your reference image and bounding boxes.
    2. Run the script to generate the ONNX file.
    3. Load the ONNX model in YoloDotNet.
    import numpy as np
from ultralytics import YOLOE
from ultralytics.models.yolo.yoloe import YOLOEVPSegPredictor

# Load the text/visual YoloE model
model = YOLOE("yoloe-11m-seg.pt")

# Define visual prompts
visual_prompts = dict(
    bboxes=np.array(
        [
            [352, 185, 375, 207] # Bounding box in [x, y, x + width, y + height] format
        ]),
    cls=np.array(
        [
            0 # Class id to be assigned for the bounding box
        ])
)

# Run prediction on a different image, using reference image to guide what to look for
results = model.predict(
    "target_image.jpg",                 # Target image for detection
    refer_image="visual_prompt.jpg",    # Reference image used to get visual prompts
    visual_prompts=visual_prompts,
    predictor=YOLOEVPSegPredictor,
    conf=0.1                            # The prompts may require a lower confidence value
)

# Show inference results
results[0].show()

# Export the model for use in YoloDotNet
model.export(format="onnx", device=0, opset=17)

  12. Install the OpenVINO execution provider

    master

    To use Intel hardware acceleration (CPU, GPU, or NPU) with YoloDotNet, you must install both the core library and the OpenVINO execution provider. Note that the OpenVINO provider requires the Intel® OpenVINO™ Runtime to be installed on your system separately.

    1. Install the core package:
    dotnet add package YoloDotNet
    1. Install the OpenVINO provider:
    dotnet add package YoloDotNet.ExecutionProvider.OpenVino

    Note: YoloDotNet.ExecutionProvider.OpenVino v1.1 requires YoloDotNet version 4.1 or higher.

    dotnet add package YoloDotNet
dotnet add package YoloDotNet.ExecutionProvider.OpenVino