DOCSEARCHbyAlgolia
Submit Docs

NVIDIA Omniverse Synthetic Data Examples

repository·main·Indexed 41 days ago

https://github.com/nvidia-omniverse/synthetic-data-examples

A collection of practical examples and workflows for generating and using synthetic data to train AI models. It leverages NVIDIA Omniverse, Omniverse Replicator, TAO Toolkit, and NGC to demonstrate end-to-end pipelines, including synthetic data generation (SDG) via Isaac Sim, model training (e.g., DetectNet_v2), and deployment to NVIDIA Jetson hardware. Includes specific labs for object detection of fruit and warehouse palletjacks.

Tokens
12.7K
Snippets
33
Records
58
Agent score
51%

What's inside Omniverse Synthetic Data Examples

  1. Overview of the Palletjack Synthetic Data Workflow

    main
    This project demonstrates an end-to-end workflow for training computer vision models using synthetic data. The workflow utilizes Isaac Sim with Omniverse Replicator to generate high-quality datasets from sim-ready assets (specifically the palletjack assets from the Warehouse collection). The generated data is exported in the KITTI format to ensure compatibility with model training. The workflow covers three main stages: Synthetic Data Generation (SDG), Model Training (using NVIDIA TAO Toolkit), and Deployment (on NVIDIA Jetson via Isaac ROS or Deepstream).

  2. Overview of Synthetic Data Examples

    main

    This repository provides examples for generating and utilizing synthetic data. The examples primarily leverage the following NVIDIA tools and ecosystems:

    • NVIDIA Omniverse: A platform for building and operating metaverse applications.
    • Omniverse Replicator: An extension for generating synthetic data for AI training.
    • NVIDIA Tao: A toolkit for accelerating the development of AI models.
    • NVIDIA NGC: A catalog of pre-trained, optimized AI models and containers.

  3. Train Models using TAO Toolkit

    main

    The workflow demonstrates training a model using the TAO Toolkit. Follow the training/cloud_train notebook for a complete walkthrough of these steps:

    1. Environment Setup: Set up the TAO Docker container.
    2. Model Selection: Download a pre-trained model. This workflow uses the DetectNet_v2 model with a resnet_18 backbone.
    3. Training: Execute TAO training using the provided spec files.
    4. Evaluation: Visualize model performance on real-world data.
    5. Monitoring: Use Tensorboard to visualize model metrics during training.

  4. Train models using NVIDIA TAO Toolkit

    main

    The workflow uses the NVIDIA TAO Toolkit for training, though users can substitute this with any preferred framework. The TAO training process involves:

    1. Generating Tfrecords from the synthetic data.
    2. Model training and evaluation, which includes:
      • Selecting a model backbone.
      • Specifying hyperparameters via a .spec file (provided in the repository).
    3. Running inference with the resulting trained model.

  5. Export a trained model with export.py

    main

    Use export.py to export your trained model. You can view all parameters by running python export.py --help.

    Important Constraints:

    • export.py only accepts checkpoints created by the current version of train.py. If you are using checkpoints from an earlier version, you must retrain the model to regenerate compatible checkpoints.
    • When exporting, ensure you save the output to the specific directory structure required, such as models/fasterrcnn_resnet50/1.

    Example command:

    python export.py -d /home/out.pth -o /home/models/fasterrcnn_resnet50/1

  6. Generate synthetic data headlessly via terminal

    main

    To generate data without a GUI, use the omni.code.replicator.sh shell script. This is useful for automated pipelines or remote servers.

    1. Locate the script: Find the local installation path of omni.code.replicator.sh within your Omniverse Code installation directory.
    2. Identify the target script: Use generate_data_headless.py located at /FruitBasketOVEReplicatorDemo/data_generation/code/generate_data_headless.py.
    3. Execute: Run the shell script with the --no-window flag and point the --/omni/replicator/script argument to your Python script.
    ./omni.code.replicator.sh --no-window --/omni/replicator/script="/FruitBasketOVEReplicatorDemo/data_generation/code/generate_data_headless.py"

  7. Start the Triton Inference Server

    main

    To deploy the model, run the Triton Inference Server using Docker. Ensure your model is located in the directory mapped to /models inside the container. For this workflow, the model should be at /models/fasterrcnn_resnet50/1 within the container's filesystem.

    Use the following command to start the server, mapping the host's model directory to the container's /models directory and exposing the necessary ports (8000, 8001, 8002):

    sudo docker run --gpus=1 --rm -p9000:8000 -p9001:8001 -p9002:8002 -v /home/$USER/sdg_workflow/models/:/models nvcr.io/nvidia/tritonserver:23.01-py3 tritonserver --model-repository=/models

    Upon successful startup, the server logs should display the model status as READY:

    +---------------------+---------+--------+
| Model               | Version | Status |
+---------------------+---------+--------+
| fasterrcnn_resnet50 | 1       | READY  |
+---------------------+---------+--------+

  9. Generate synthetic data using Isaac Sim and Replicator

    main

    To generate the dataset, use Isaac Sim and Omniverse Replicator to perform Domain Randomization. This process involves modifying various scene attributes to ensure the model generalizes well to the real world, including:

    • Lighting
    • Textures
    • Object pose
    • Materials

    Data Output: The annotations are generated using the KITTI Writer. While the default is KITTI format, you can implement a custom writer if your specific model requires a different data format.

  11. Generate Synthetic Data for Palletjack Objects

    main

    To generate synthetic data for warehouse palletjack objects in KITTI format:

    1. Container: Use the Isaac Sim Docker container to run the data generation script.
    2. Script: Execute the data generation script located at ../palletjack_sdg/palletjack_datagen.sh.
    3. Guidance: Follow the step-by-step instructions provided in the cloud_sdg notebook.

    The resulting data is compatible with various training frameworks; this specific workflow uses the TAO Toolkit.

    # Run the data generation script using the Isaac Sim docker container
../palletjack_sdg/palletjack_datagen.sh

  12. Build TensorRT engines (FP16 and INT8)

    main

    Convert the downloaded ONNX model into a TensorRT engine using one of the following methods:

    Option 1: Build FP16 engine (Recommended)

    Use this for a balance of accuracy and performance.

    Option 2: Build INT8 engine

    Note: Current INT8 instructions do not include calibration. Use this only for throughput profiling, as accuracy may vary significantly from FP32/FP16 models.

    Both options require the <onnx_path> (path to your downloaded ONNX file) and <engine_output_path> (where you want to save the resulting engine).

    # Build FP16 engine (Recommended)
./build_trt_fp16.sh <onnx_path> <engine_output_path>

# Build INT8 engine (For profiling only)
./build_trt_int8.sh <onnx_path> <engine_output_path>