DOCSEARCHbyAlgolia
Submit Docs

jetson-inference

repository·master·Indexed 27 days ago

https://github.com/dusty-nv/jetson-inference

A library and instructional suite for deploying optimized deep learning models on NVIDIA Jetson hardware using TensorRT and PyTorch. It provides high-level C++ and Python APIs for vision primitives including image classification (imageNet), object detection (detectNet), semantic segmentation (segNet), pose estimation (poseNet), action recognition (actionNet), background removal (backgroundNet), and monocular depth estimation (depthNet). Includes the jetson-utils library for image processing and hardware interfacing.

Tokens
55.5K
Snippets
166
Records
279
Agent score
94%

What's inside jetson-inference

  1. Overview of jetson-inference

    master

    jetson-inference is an instructional guide and library for real-time vision and deep learning inference on NVIDIA Jetson devices. It uses TensorRT to run optimized networks on GPUs via C++ or Python, and PyTorch for model training.

    Supported vision primitives include:

    • imageNet: Image classification
    • detectNet: Object detection
    • segNet: Semantic segmentation
    • poseNet: Pose estimation
    • actionNet: Action recognition
    • backgroundNet: Background removal
    • depthNet: Monocular depth estimation

  2. Understand the Deep Learning Workflow: Training vs Inference

    master

    Deep learning development consists of two primary phases:

    1. Training: The process where a network learns from a large dataset of labeled examples. Weights are optimized to recognize patterns. This phase is resource-intensive and benefits significantly from GPU acceleration.
    2. Inference: The process where a trained network uses its weights to evaluate live data at runtime. It predicts and applies reasoning based on learned patterns. On Jetson platforms, inference is performed using NVIDIA's GPU Inference Engine to enable real-time processing on embedded hardware.

    Common inference applications include:

    • Image recognition
    • Object detection
    • Segmentation
    • Image registration (homography estimation)
    • Depth from raw stereo
    • Signal analytics

  3. Use backgroundNet for background removal and replacement

    master
    The backgroundNet object uses a U²-Net fully-convolutional network to generate a foreground mask that segments the foreground from the background. This can be used to replace or blur backgrounds or as a pre-processing step for other vision tasks like object detection or motion detection. It is available for both C++ and Python implementations.

  4. Use SegNet for Semantic Segmentation

    master

    segNet performs semantic segmentation by providing per-pixel classification masks. It accepts a 2D image as input and outputs an image with a classification mask overlay. It is available for use in both C++ and Python.

    Key features:

    • Performs per-pixel labeling (Fully Convolutional Network).
    • Supports images, videos, and camera feeds.
    • Available via the segNet class in C++ and Python.

  5. Understand DetectNet Object Detection

    master
    The detectNet object performs object detection and localization, meaning it identifies objects in a 2D image and extracts their bounding box coordinates. It is built upon a pretrained ImageNet recognition model (such as Googlenet) and uses bounding coordinate labels in the training dataset to learn object locations.

  6. Choose a method to use jetson-inference

    master

    There are two primary ways to use the jetson-inference project:

    1. Run the pre-built Docker Container: Recommended for getting started quickly, as the container already includes PyTorch and other dependencies.
    2. Build the Project from Source: Recommended if you prefer native development and want to compile the project yourself.

  7. Explore NVIDIA Jetson and AI IoT resources

    master

    Additional resources for Jetson development and NVIDIA AI IoT projects can be found in the following repositories and wikis:

    • NVIDIA AI IoT: Official NVIDIA Jetson GitHub repositories.
    • Jetson eLinux Wiki: Community and technical wiki for Jetson hardware and software.

  8. Configure and transmit RTP streams

    master

    RTP streams are broadcast over UDP/IP.

    Receiving RTP: You MUST explicitly specify the codec using --input-codec because RTP cannot dynamically query it. Use rtp://@:PORT for localhost or rtp://<multicast-group>:PORT for multicast.

    Transmitting RTP: Specify the target IP and port as the output_URI. You can control the bitrate with --bitrate (default 4Mbps) and the codec with --output-codec (h264, h265, vp8, vp9, mjpeg).

    # Receiving RTP
$ video-viewer --input-codec=h264 rtp://@:1234         # receive on localhost port 1234
$ video-viewer --input-codec=h264 rtp://224.0.0.0:1234 # subscribe to multicast group

# Transmitting RTP
$ video-viewer --bitrate=1000000 csi://0 rtp://<remote-ip>:1234         # transmit camera over RTP, encoded as H.264 @ 1Mbps 
$ video-viewer --output-codec=h265 my_video.mp4 rtp://<remote-ip>:1234  # transmit a video file over RTP

  9. Install JetPack on Jetson TX1/TX2, AGX Xavier, and AGX Orin

    master

    For other Jetson models (such as TX1/TX2, AGX Xavier, and AGX Orin), use the NVIDIA SDK Manager on a host PC running Ubuntu x86_64.

    Steps:

    1. Connect the Micro-USB or USB-C port of the Jetson to your host PC.
    2. Enter the Jetson device into Recovery Mode.
    3. Use the SDK Manager to flash the device.

    Refer to the NVIDIA SDK Manager documentation for detailed instructions.

  10. Download the PlantCLEF Dataset

    master

    To perform plant classification training, download and extract the PlantCLEF subset dataset. Ensure you are in the jetson-inference/python/training/classification/data directory before running the commands.

    $ cd jetson-inference/python/training/classification/data
$ wget https://nvidia.box.com/shared/static/vbsywpw5iqy7r38j78xs0ctalg7jrg79.gz -O PlantCLEF_Subset.tar.gz
$ tar xvzf PlantCLEF_Subset.tar.gz

  11. Run backgroundNet CLI examples

    master

    You can use the compiled backgroundnet (C++) or backgroundnet.py (Python) binaries to process images or live camera streams via the command line.

    Image Processing:

    • To remove the background (outputting an image with an alpha mask): [executable] <input_image> <output_mask_image>
    • To replace the background: [executable] --replace=<replacement_image> <input_image> <output_image>

    Live Streaming: Pass a video device (e.g., /dev/video0) instead of an image file to process a live stream.

    Arguments:

    • --replace=<filename>: Specifies an image to replace the background with. The replacement image is automatically re-scaled to match the input resolution.
    # C++
$ ./backgroundnet images/bird_0.jpg images/test/bird_mask.png                                 # remove the background (with alpha)
$ ./backgroundnet --replace=images/snow.jpg images/bird_0.jpg images/test/bird_replace.jpg    # replace the background

# Python
$ ./backgroundnet.py images/bird_0.jpg images/test/bird_mask.png                              # remove the background (with alpha)
$ ./backgroundnet.py --replace=images/snow.jpg images/bird_0.jpg images/test/bird_replace.jpg # replace the background

# Live Streaming (C++ or Python)
$ ./backgroundnet /dev/video0                             # remove the background
$ ./backgroundnet --replace=images/coral.jpg /dev/video0  # replace the background

  12. Check Open Images dataset statistics

    master

    Before downloading large datasets, use the --stats-only flag to see the number of available images and bounding boxes for your selected classes. This helps estimate disk space and training time without downloading the actual images (though annotation data will be downloaded, ~1GB).

    python3 open_images_downloader.py --stats-only --class-names "Apple,Orange,Banana,Strawberry,Grape,Pear,Pineapple,Watermelon" --data=data/fruit