NanoOWL Documentation

To use NanoOWL on NVIDIA Jetson Orin platforms, follow these steps to install dependencies, the package itself, and build the required TensorRT engine for the OWL-ViT vision encoder.

1. Install Dependencies

Ensure you have installed:

• PyTorch
• torch2trt
• NVIDIA TensorRT
• transformers library via python3 -m pip install transformers

2. Install NanoOWL

Clone the repository and install the package in development mode:

git clone https://github.com/NVIDIA-AI-IOT/nanoowl
cd nanoowl
python3 setup.py develop --user

3. Build the TensorRT Engine

You must build the engine for the vision encoder before running predictions:

mkdir -p data
python3 -m nanoowl.build_image_encoder_engine data/owl_image_encoder_patch32.engine

# Install dependencies
python3 -m pip install transformers

# Install NanoOWL
git clone https://github.com/NVIDIA-AI-IOT/nanoowl
cd nanoowl
python3 setup.py develop --user

# Build the engine
mkdir -p data
python3 -m nanoowl.build_image_encoder_engine data/owl_image_encoder_patch32.engine

NanoOWL's "tree prediction" pipeline combines OWL-ViT and CLIP to enable nested detection and classification. You can define hierarchical relationships or simple classifications using a specific prompt syntax.

Prompt Syntax Patterns:

• Nested Detection: Use square brackets [] to define sub-objects within a detected object.
  • Example: "[an owl [a wing, an eye]]" (Detect an owl, then detect wings and eyes inside the owl regions).
• Classification: Use parentheses () to provide mutually exclusive classes.
  • Example: "(indoors, outdoors)".
• Combined Logic: Combine classification and nested detection.
  • Example: "(indoors, outdoors [an owl])" (Classify as indoors/outdoors; if outdoors, detect owls).

CLI Usage:

Run tree_predict.py from the examples directory:

python3 tree_predict.py \
    --prompt="[an owl [a wing, an eye]]" \
    --threshold=0.15 \
    --image_encoder_engine=../data/owl_image_encoder_patch32.engine

# Nested detection example
python3 tree_predict.py \
    --prompt="[an owl [a wing, an eye]]" \
    --threshold=0.15 \
    --image_encoder_engine=../data/owl_image_encoder_patch32.engine

# Classification example
python3 tree_predict.py \
    --prompt="(indoors, outdoors)" \
    --threshold=0.15 \
    --image_encoder_engine=../data/owl_image_encoder_patch32.engine

The OwlPredictor class allows you to perform open-vocabulary object detection using a TensorRT optimized OWL-ViT model. You provide a PIL Image and a list of text labels to detect.

Parameters:

• model_name: The HuggingFace model path (e.g., "google/owlvit-base-patch32").
• image_encoder_engine: Path to the built .engine file.
• predict(image, text, threshold):
  • image: A PIL Image object.
  • text: A list of strings representing the objects to detect.
  • threshold: Confidence threshold for detection.

from nanoowl.owl_predictor import OwlPredictor
import PIL.Image

predictor = OwlPredictor(
    "google/owlvit-base-patch32",
    image_encoder_engine="data/owlvit-base-patch32-image-encoder.engine"
)

image = PIL.Image.open("assets/owl_glove_small.jpg")
output = predictor.predict(image=image, text=["an owl", "a glove"], threshold=0.1)

print(output)