DOCSEARCHbyAlgolia
Submit Docs

Perception Package

repository·main·Indexed 21 days ago

https://github.com/unity-technologies/com.unity.perception

A toolkit for generating large-scale synthetic datasets for training and validating computer vision models within the Unity engine. Version 1.0.0-preview.1 supports labeling, domain randomization, and output formats such as SOLO and COCO. Key features include the Perception Camera for RGB and ground truth capture, Dataset Capture for sensor orchestration, and support for HDRP Path Tracing via accumulation settings.

Tokens
41K
Snippets
73
Records
193
Agent score
76%

What's inside com.unity.perception

  1. Overview of Perception Fundamentals

    main

    The Perception package provides tools for generating synthetic datasets for machine learning. Core components include:

    • Perception Camera: Captures RGB images and ground truth data from a Unity Camera.
    • Output Endpoints: Defines where data is sent. Supported endpoints are No Output, Perception (legacy), and Solo (current standard).
    • Labeling: Uses components to mark GameObjects and descendants with specific labels based on a Label Config (a taxonomy asset).
    • Randomization: A toolset for integrating domain randomization into simulations via Scenarios, Randomizers, and Samplers.
    • Data Generation: Orchestrates the capture process using Dataset Capture to ensure sensors trigger at correct rates and data is formatted into JSON datasets.

  2. Explore Perception sample projects and datasets

    main

    Unity provides several specialized projects and datasets built with or around the Perception ecosystem:

    • Synthetic Homes: A 100K image dataset of annotated synthetic home interiors and a standalone Unity application for generation.
    • Synthetic Humans: A package for procedurally generating and placing diverse groups of synthetic humans.
    • People Sans People: A privacy-preserving synthetic data generator with highly parameterized domain randomization (supports 2D/3D bounding boxes, instance/semantic segmentation, and COCO pose labels).
    • SynthDet: An end-to-end solution for training 2D object detection models using synthetic data.
    • Robotics Object Pose Estimation Demo: Demonstrates pick-and-place with a robot arm, including ROS integration and URDF model importing.

  3. Use post-processing effects safely in synthetic datasets

    main

    When using post-processing effects for computer vision (CV) training, you must ensure they do not introduce non-determinism or insufficient randomization.

    Risks

    • Insufficient Randomization: Effects like film grain may use internal noise patterns that are not sufficiently randomized, causing models to learn specific noise patterns rather than general features.
    • Non-determinism: If an effect uses random number generators (RNGs) outside of the Perception package's Samplers, you will be unable to reproduce your datasets.

    Best Practices

    • Use Perception Samplers: If you implement or modify a randomized effect, ensure it only uses the Samplers provided by the Perception package to maintain determinism.
    • Recommended Effects: Experiments suggest that randomizing contrast, saturation, lens blur, and lens distortion can significantly improve CV model performance.

  4. Understand the role of Scenarios

    main

    In the Perception package, a Scenario is a high-level controller responsible for three main tasks:

    1. Execution Flow: Controlling how the simulation runs.
    2. Organization: Managing a list of Randomizers.
    3. Configuration: Defining settings that can be modified externally from a built Unity player via JSON.

    The package provides a default implementation called FixedLengthScenario, which runs each iteration for a fixed number of frames to generate datasets of a specific size.

  6. How Randomizers and Tags work together

    main

    The Perception package uses a two-part system for randomization:

    1. Randomizers: These are components added to the Scenario. They act as the controllers that drive the randomization logic for the entire simulation.
    2. Tags: These are components added to specific GameObjects in the scene. They act as targets for the Randomizers and hold the specific configuration (like ranges or lists) for that individual object.

    This separation allows a single Randomizer to apply different randomization logic to different objects based on their unique Tag configurations. For example, a LightRandomizer in the Scenario can randomize multiple lights differently if each light has a LightRandomizerTag with different color or intensity ranges.

  7. Understand the core components of the Perception randomization framework

    main

    The Perception randomization toolset is built on five primary abstractions that work together to create variability in synthetic datasets:

    • Scenarios: The central controller. It manages the execution flow, holds a list of Randomizers, and defines constants that can be configured externally. Each Iteration of a Scenario represents one complete run of the environment under a unique set of randomized conditions.
    • Randomizers: The workers. They encapsulate specific tasks (like varying material properties or repositioning lights) and expose Parameters to the Inspector.
    • Randomizer Tags: The selectors. RandomizerTags allow Randomizers to query and identify specific subsets of GameObjects to act upon (e.g., a rotation randomizer looking for objects with a specific RotationRandomizerTag).
    • Parameters: The translators. Parameters use Samplers to generate random values and transform them into usable C# types (e.g., converting float samples into a Vector3 or a Material).
    • Samplers: The generators. Samplers produce bounded random float values based on specific probability distributions (defined by a minimum and maximum value).

  8. Use the Perception Camera component for deterministic data capture

    main

    The Perception Camera component is used to ensure a Unity Camera runs at deterministic rates and integrates with DatasetCapture to save RGB and ground truth data into a JSON dataset. It is compatible with the High Definition Render Pipeline (HDRP).

    Core Properties

    • Description: A string describing the camera for the JSON dataset.
    • Show Visualizations: Toggles real-time visualizations for active labelers.
    • Capture RGB Images: When enabled, captures RGB images as PNG files each frame.
    • Capture Trigger Mode: Determines how captures are triggered:
      • Scheduled: Automatic captures based on a start frame and frame delta time.
      • Manual: Requires calling the RequestCapture method via script.
    • Override Layer Mask: If enabled, labelers use the Layer Mask property on the Perception Camera instead of the standard camera culling mask. This is useful for controlling object visibility during segmentation rendering.
    • Sensor Type: Selects the Camera Sensor used for pixel-based ground truth.
    • Camera Labelers: A list of labelers (e.g., Semantic Segmentation, Bounding Box) that generate data from this camera.
    # Example of triggering a manual capture via script
// Note: This assumes you have a reference to the PerceptionCamera component
perceptionCamera.RequestCapture();

  9. What are Samplers in Perception

    main

    Samplers are classes used to deterministically generate random float values from bounded probability distributions. While they are frequently used with Parameters to create arrays of typed random values, they can be instantiated and used independently from any standard C# script.

    To ensure deterministic simulations (consistent results across local execution and cloud simulation), you should use Perception Samplers instead of UnityEngine.Random or System.Random, as the latter can lead to non-deterministic behavior. Samplers are seeded by the active Scenario's current random state.

  11. Ensure deterministic datasets with Random Seeds

    main

    Using the same Random Seed in two runs of the same Scenario guarantees identical datasets only if:

    1. You exclusively use Perception package Samplers and Parameters.
    2. The order of sampling remains identical (e.g., you do not change the order of Randomizers in the Scenario or add/remove Sampler calls).
    3. The number of samples obtained remains identical.

    Warning: Certain Unity systems are non-deterministic and can cause variations even with the same seed. These include:

    • The Physics engine
    • Film grain post-processing

  12. Configure the Unity Camera Sensor and Super Sampling

    main

    The Unity Camera Sensor is the default sensor. It captures data from the camera component configured on the Perception Camera's GameObject.

    Super Sampling Factor

    You can use the Super Sampling Factor to perform high-quality Super Sampling Anti-Aliasing (SSAA). This uses sub-pixel averaging to smooth edges and improve texture detail.

    • Available Factors: 2, 4, or 8.
    • Resolution Impact: The output resolution of the sensor does not change; however, the internal capture resolution increases (e.g., a factor of 4 results in 16x the total pixels).
    • Warning: Increasing the Super Sampling Factor increases RAM usage and impacts performance.