DOCSEARCHbyAlgolia
Submit Docs

Roboflow Inference

repository·main·Indexed 25 days ago

https://github.com/roboflow/inference

A platform to turn computers or edge devices into AI cameras, allowing users to self-host fine-tuned models, access foundation models, and build computer vision pipelines using composable Workflows. It includes the inference-models engine (v0.34.3) and supports deployment via Docker across x86 CPU, GPU, ARM, and Jetson architectures, as well as AWS Lambda.

Tokens
224.8K
Snippets
559
Records
973
Agent score
81%

What's inside roboflow-inference

  1. Overview of Roboflow Inference features

    main

    Roboflow Inference is a comprehensive computer vision platform designed for model serving, video stream management, and complex computation graphs. Key capabilities include:

    • Model Serving: Supports Object Detection, Image Classification, Instance Segmentation, Keypoint Detection, Image Embedding, OCR, and Visual Question Answering.
    • Workflows: Uses a computation graph of 'Workflow Blocks' to pipe and parallelize data through models, logic, and custom code.
    • Video Stream Management: Uses separate threads for image ingestion to ensure models always receive the most recent frame without hanging.
    • Deployment & Portability: Can be deployed via Docker across various platforms, from MacOS development machines to edge devices and cloud servers.
    • Offline Capabilities: Supports an offline mode by caching models and Workflow definitions locally.
    • Optimization: Features automatic parallelization, hardware acceleration, and dynamic batching. Supports a TensorRT flag for quantization and device-specific layer fusion on supported GPUs.

  2. Overview of Roboflow Enterprise Inference features

    main

    Roboflow Enterprise customers have access to advanced Inference capabilities designed for production-scale computer vision. These features include:

    • Active learning: Tools to actively collect data from production environments to facilitate training of more accurate models over time.
    • Parallel processing server: A server configuration that allows running requests in parallel to increase throughput and reduce latency during model inference.
    • Multi-device licensing: A license that permits running Inference on more than one device.

    For information regarding enterprise offerings, contact the Roboflow sales team.

  3. What is inference-models?

    main

    inference-models is a library designed to make predictions from computer vision models provided by Roboflow. It features:

    • Multi-Backend Support: Supports PyTorch, ONNX, TensorRT, and Hugging Face.
    • Automatic Model Loading: Handles model resolution and backend selection automatically.
    • Minimal Dependencies: Uses a composable extras system to keep installations lightweight.
    • Consistent APIs: Models with similar behaviors share consistent interfaces.
    • Roboflow Integration: Full support for any model trained on the Roboflow platform.

  4. Compare Roboflow Managed Compute options

    main

    Roboflow offers three primary ways to deploy Inference, depending on your requirements for infrastructure management, model complexity, and data privacy:

    1. Serverless Hosted API: Best for quick starts and scaling without managing infrastructure. It supports Workflows, model chaining, and cloud-hosted VLMs (ChatGPT, Claude), but does not support video streaming or heavy models like Florence-2/SAM 2.
    2. Dedicated Deployments: Single-tenant virtual machines for high-performance needs. Supports video streaming, custom Python code, heavy foundation models (SAM 2, Florence-2, PaliGemma), and additional dependencies. Available in Development mode (3-hour sessions, potential eviction) or Production mode (guaranteed capacity).
    3. Bring Your Own Cloud (BYOC): For enterprise compliance and sensitive data requirements, allowing you to self-host workloads on your own cloud infrastructure.

  6. Depth Anything V2 - Monocular Depth Estimation

    main

    Depth Anything V2 is a monocular depth estimation model that uses a vision transformer architecture to predict relative depth maps from single RGB images. It offers multiple model sizes to balance speed and accuracy and features zero-shot generalization across diverse image types.

    Key Capabilities:

    • Monocular Depth Estimation: Predict depth from a single image.
    • Multiple Model Sizes: Choose between small, base, or large variants.
    • High Quality: State-of-the-art performance on depth estimation benchmarks.

  7. Deployment options for the Inference Server

    main

    The Inference Server is a standalone microservice that wraps the inference Python package and exposes it via an HTTP API. You can deploy it using several paths:

    • Self-Hosted: Run on your own edge hardware (e.g., Raspberry Pi, NVIDIA GPU, NVIDIA Jetson) using Docker.
    • Deploy in Your Own Cloud: Run on your own cloud infrastructure (AWS, GCP, Azure) using Docker.
    • Serverless API: Hosted by Roboflow, scales to zero, and uses a pay-per-inference model.
    • Dedicated Deployments: Hosted by Roboflow on single-tenant VMs with optional GPU support.

    You can interact with any of these deployment options using the Inference SDK.

  8. Hardware compatibility for inference-models

    main

    The inference-models package supports various hardware configurations including x86_64 CPUs, Apple Silicon, NVIDIA GPUs, and NVIDIA Jetson devices.

    Support Summary

    • CPU (x86_64): Stable on Linux and macOS (Bare-metal or Docker).
    • Apple Silicon: Stable on macOS (Bare-metal or Docker). MPS (Metal Performance Shaders) acceleration is experimental and limited to select models like RFDetr.
    • NVIDIA GPU: Stable on Ubuntu 22.04/24.04 LTS (Bare-metal or Docker). Requires CUDA 11.8 or 12.x and NVIDIA Container Toolkit for Docker.
    • NVIDIA Jetson:
      • JetPack 6.1: Stable via Docker; Experimental via Bare-metal.
      • JetPack 5.1: Stable via custom Docker build; Bare-metal is not possible.
    • Windows: Experimental. Requires elevated admin access or Developer Mode enabled for cache management (symlinks).

  9. Use PaliGemma for Vision-Language Tasks

    main

    PaliGemma and PaliGemma2 are vision-language models (VLMs) that combine a SigLIP vision encoder with a Gemma language model. They can perform various tasks including Visual Question Answering (VQA), Image Captioning, Object Detection, OCR, Document Understanding, and Counting via natural language prompts.

    Important Requirements:

    • GPU Recommended: CPU inference may be extremely slow or non-functional. Use a GPU for acceptable performance.
    • Resolution Trade-off: Use 224px models for speed and 448px models for higher accuracy/quality.

  10. Explore Roboflow Inference project examples

    main

    The examples/ directory contains various implementations of projects built using the Roboflow Inference engine. These range from simple clients to complex stream processing and search engines:

    • Inference Client: Quickstart HTTP and UDP clients for interacting with Inference services.
    • Inference Dashboard: A tool to extract insights from video frames at defined intervals, generating visualizations and CSV outputs.
    • CLIP Image-to-Image Search Engine: A web application that uses CLIP to find images in a dataset similar to a provided input image.
    • SAM Client: A command-line application that uses the Segment Anything Model (SAM) to segment objects within an image.
    • Stream Examples: Implementations for running computer vision models on webcam and RTSP streams, including examples using Roboflow-hosted models and locally deployed models with Inference and CLIP.

  11. Supported Foundation Models in Roboflow Inference

    main

    Roboflow Inference supports several foundation models that can be accessed via HTTP requests, eliminating the need for manual local configuration of each model. Supported models include:

    • CLIP: Used for image classification and comparing the similarity between images and text.
    • DocTR: Used for reading characters within images (OCR).
    • Grounding DINO: Enables zero-shot object detection using text prompts. It can also be used to automatically label data for training smaller, specialized models.
    • Segment Anything 3 (SAM3): Used for segmenting objects in images.

    Note: Gaze (L2CS-Net) is deprecated and no longer functional.

    All models are accessible through the Inference API, allowing you to perform vision tasks like clustering, moderation, and segmentation without deep machine learning expertise.

  12. What is a Sink in an Inference Pipeline?

    main

    A Sink is a function that defines what the InferencePipeline should do with each prediction. It is the logic that executes after inference is complete (e.g., rendering boxes, saving to a file, or broadcasting data).

    Sinks are injected into the InferencePipeline.init() method via the on_prediction parameter.

    Sink Signature

    A sink must follow this signature to handle both single and multiple video sources:

    from typing import Union, List, Optional
from inference.core.interfaces.camera.entities import VideoFrame

def on_prediction(
    predictions: Union[dict, List[Optional[dict]]],
    video_frame: Union[VideoFrame, List[Optional[VideoFrame]]],
) -> None:
    # If using multiple sources or batch mode, iterate through the lists
    # Note: elements can be None if a source disconnected
    for prediction, frame in zip(predictions, video_frame):
        if prediction is None:
            continue
        # Process prediction and frame
    from typing import Union, List, Optional
from inference.core.interfaces.camera.entities import VideoFrame

def on_prediction(
    predictions: Union[dict, List[Optional[dict]]],
    video_frame: Union[VideoFrame, List[Optional[VideoFrame]]],
) -> None:
    for prediction, frame in zip(predictions, video_frame):
        if prediction is None:
            # EMPTY FRAME
            continue
        # SOME PROCESSING