DOCSEARCHbyAlgolia
Submit Docs

usls

repository·main·Indexed 19 days ago

https://github.com/jamjamjon/usls

A high-performance Rust library for vision and vision-language model inference integrated with ONNX Runtime. It provides cross-platform support for Linux, macOS, and Windows with hardware acceleration via CUDA, TensorRT, and CoreML. The library includes a comprehensive viz module for visual annotations, supporting horizontal and oriented bounding boxes, keypoints, polygons, segmentation masks, and classification probabilities.

Tokens
63.1K
Snippets
193
Records
238
Agent score
64%

What's inside usls

  1. Overview of the usls library

    main

    usls is a cross-platform Rust library designed for efficient inference of State-of-the-Art (SOTA) vision and vision-language models (VLM). It is powered by ONNX Runtime and provides a unified interface for high-performance machine learning workflows.

    Key Capabilities

    • High Performance: Utilizes multi-threading, SIMD, and CUDA-accelerated processing via ONNX Runtime execution providers.
    • Cross-Platform Support: Works on Linux, macOS, and Windows with support for various hardware accelerators including CUDA, TensorRT, CoreML, OpenVINO, and DirectML.
    • Unified API: All models implement a single Model trait featuring run()/forward() methods and a unified Y output format.
    • Auto-Management: Automatically handles model downloads from HuggingFace or GitHub, including caching and path resolution.
    • Flexible Inputs: Supports images, directories, videos, webcams, and streams.
    • Precision Support: Offers various quantization options such as FP32, FP16, INT8, Q4, Q4F16, and BNB4.
    • Full-Stack Suite: Includes complete workflows with DataLoader, Annotator, and Viewer components.

  2. Explore the Model Zoo categories

    main

    The Model Zoo contains various computer vision models organized by task. Available categories include:

    • YOLO Series: YOLOv5 through YOLO13, and YOLO26.
    • Classification & Tagging: Image classification and tagging.
    • Object Detection: DETR series, PicoDet, D-FINE, DEIM.
    • Image Segmentation: SAM series, FastSAM, YOLOE, BiRefNet.
    • Background Removal: RMBG, BEN2.
    • Gaze Estimation: MobileGaze.
    • Image Matting: MODNet, MediaPipe Selfie, BiRefNet variants.
    • Open-Set Detection: GroundingDINO, MM-GDINO, LLMDet, OWLv2, YOLO-World.
    • Multi-Object Tracking: ByteTrack.
    • Super Resolution: Swin2SR, APISR.
    • Pose Estimation: RTMPose, DWPose, RTMW, RTMO.
    • OCR & Documents: Text detection/recognition, table recognition, document layout.
    • Vision-Language Models: BLIP, Florence2, Moondream2, SmolVLM, FastVLM.
    • Embedding Models: CLIP, jina-clip, MobileCLIP, DINOv2/v3.
    • Depth Estimation: DepthAnything, DepthPro.
    • Others: Sapiens, YOLOPv2.

  3. Understand the core concepts of usls

    main

    The usls library is organized around several key pillars for model execution and data processing:

    • Module System: Models are organized into specific functional modules: Model, Visual, Textual, Encoder, and Decoder.
    • Configuration: Uses a builder-pattern API to configure models with specific devices, data types (dtypes), and execution providers.
    • Data Loading: Provides efficient loading for images, videos, webcams, and streams, featuring automatic batching.
    • Results & Annotation: Emits unified Y results. Visualization and manual annotation are handled via the Annotator and Viewer components.
    • Execution Providers: Supports hardware acceleration via providers like CUDA, TensorRT, CoreML, OpenVINO, and DirectML.
    • Device Management: Allows for independent configuration of model and processor devices to optimize performance.
    • Data Types: Supports multiple precision levels including FP32, FP16, INT8, Q8, Q4F16, and BNB4.

  4. What is SAM3-LiteText text-encoder ONNX export?

    main

    SAM3-LiteText is a variant of SAM3-Image where the heavy text encoder is replaced by a distilled MobileCLIP student, while keeping the vision encoder, geometry encoder, and mask decoder intact.

    This tool exports the lightweight text encoder to ONNX format. The exported model is a drop-in replacement for the standard SAM3 text encoder with the following interface:

    • Inputs:
      • input_ids: shape [B, 32]
      • attention_mask: shape [B, 32]
    • Outputs:
      • text_features: shape [B, 32, 256]
      • text_mask: shape [B, 32]

    Available variants:

    variantHF modeltext encoder
    s0vil-uob/sam3-litetext-s0MobileCLIP-S0
    s1vil-uob/sam3-litetext-s1MobileCLIP-S1
    lvil-uob/sam3-litetext-lMobileCLIP2-L

  5. How model and processor devices work together

    main

    In usls, device management is split between two distinct components: the Model and the Image Processor.

    • Model: Handles neural network inference (e.g., Device::Cuda(0), Device::TensorRT(0)).
    • Image Processor: Handles image preprocessing tasks like resizing and normalization (e.g., Device::Cpu, Device::Cuda(0)).

    You can run these on different devices to optimize performance. For example, you can perform preprocessing on the CPU while the model runs on a GPU, or perform both on the GPU to minimize data transfer overhead.

    | Component | Description | Example Device |
| :--- | :--- | :--- |
| **Model** | Neural network inference | `Device::Cuda(0)`, `Device::TensorRT(0)` |
| **Image Processor** | Image preprocessing (resize, normalize) | `Device::Cpu`, `Device::Cuda(0)` |

  6. How the Config System works in usls

    main

    The Config System uses a builder pattern to configure models and their components. It follows a strict naming convention to make the API discoverable.

    API Naming Convention

    1. Per-Module:
      • with_<module_name>_<field_name>(<value>)
      • with_module_<field_name>(<module_name>, <value>)
    2. Global:
      • with_<field_name>_all(<value>)

    Important: Finalizing Configuration

    Always call .commit()? at the end of your configuration chain. The configuration is only validated and finalized when .commit()? is invoked.

    let config = Config::yolo()
    .with_model_device(Device::Cuda(0))
    .commit()?;
    // Single-module model
let config = Config::yolo()
    .with_model_device(Device::Cuda(0))
    .commit()?;

  7. Understand Hub caching behavior

    main

    The Hub utility implements the following caching logic:

    • Local Storage: Files are cached locally (typically in ~/.cache/usls/) after the first download.
    • Metadata TTL: GitHub release metadata is cached with a Time-To-Live (TTL). The default is 10 minutes, but it can be configured using the .with_ttl() method.
    • Atomic Writes: To prevent corruption, downloads use temporary files; failed or incomplete downloads are discarded.

  8. Avoid data transfer overhead in multi-GPU environments

    main

    When using multiple GPUs, ensure that both the model and the processor are assigned to the same GPU ID. Using different IDs for the model and processor will cause significant data transfer overhead between devices.

    // CORRECT: Both on GPU 0
config.with_model_device(Device::Cuda(0))
      .with_processor_device(Device::Cuda(0));

// INCORRECT: Different GPUs cause data transfer overhead
config.with_model_device(Device::Cuda(0))
      .with_processor_device(Device::Cuda(1));

  9. How the Annotator architecture works

    main

    The Annotator is composed of several specialized style modules, each responsible for a specific type of visual annotation. These modules are organized as follows:

    • hbb_style: HbbStyle (Horizontal bounding box)
    • obb_style: ObbStyle (Oriented bounding box)
    • keypoint_style: KeypointStyle (Keypoints & skeleton)
    • polygon_style: PolygonStyle (Polygon/contour)
    • mask_style: MaskStyle (Segmentation mask)
    • prob_style: ProbStyle (Classification probs)
    • text_renderer: TextRenderer (Font & text rendering)

  11. Understand the unified Y structure for model results

    main

    All models in the usls ecosystem return results using a unified Y structure. This structure acts as a container for various types of detection and segmentation data, including bounding boxes, masks, polygons, and keypoints. This uniformity allows you to write generic post-processing logic that works across different models.

    Key fields in the Y structure include:

    • hbbs: Vec<Hbb> (Horizontal bounding boxes)
    • obbs: Vec<Obb> (Oriented bounding boxes)
    • masks: Vec<Mask> (Segmentation masks)
    • polygons: Vec<Polygon> (Contours)
    • keypoints: Vec<Keypoint> (Keypoints)
    • keypointss: Vec<Vec<Keypoint>> (Multiple keypoint sets)
    • probs: Vec<Prob> (Classification probabilities)
    • texts: Vec<Text> (OCR/VLM text)
    • embedding: X (Feature embeddings)
    • extra: HashMap<String, X> (Model-specific data)
    • images: Vec<Image> (Output images)
    let ys: Vec<Y> = model.run(&images)?;

  12. How DataLoader works and its key features

    main

    The DataLoader is used for efficient, batched data ingestion from multiple sources with automatic memory management. It supports multi-source inputs (images, videos, webcams, URLs, directories, and globs), automatic batch collation with padding, background thread streaming for non-blocking iteration, and built-in progress bars for long-running operations.

    // Example of initializing a DataLoader with multiple sources
let dl = DataLoader::new(vec![
    "./images/*.jpg",
    "./videos/sample.mp4",
    "https://example.com/image.png",
    "0",  // Webcam
])?;