NVIDIA Isaac GR00T

NVIDIA Isaac GR00T N1.7 is an open vision-language-action (VLA) model designed for generalized humanoid robot skills. It is a cross-embodiment model that accepts multimodal inputs, including language and images, to perform manipulation tasks. The architecture combines a vision-language foundation model with a diffusion transformer (DiT) head that denoises continuous actions.

Key features of N1.7 include:

• Relative EEF Action Space: Uses relative end-effector (EEF) action deltas rather than absolute targets to improve generalization across different robot and human embodiments.
• Human Video Pretraining: Leverages 20K hours of EgoScale human video data to transfer manipulation priors to robot control.
• New VLM Backbone: Uses Cosmos-Reason2-2B (Qwen3-VL architecture) which supports flexible resolution and native aspect ratio encoding without padding.

GR00T supports two inference architectures depending on your hardware setup:

1. Direct Gr00tPolicy usage: Use this when model inference and robot control are running on the same physical machine.
2. ZMQ Server-Client architecture: Recommended for real-world deployment. This decouples the Local Client (robot-side control) from the Model Server (remote inference). This allows you to run compute-intensive inference on a GPU server while keeping the robot-side code lightweight and avoiding the need to install the full inference dependency stack on the robot.

A typical GR00T real-world deployment follows these eight stages:

1. Hardware Preparation: Verify robot platform, sensors, and compute resources.
2. Data Collection: Use teleoperation to collect at least 100 valid episodes.
3. Data Preprocessing: Clean, align timestamps, and convert to LeRobot format.
4. Model Training: Fine-tune GR00T N1.*
5. Model Evaluation: Run open-loop evaluation to validate convergence.
6. Deployment Setup: Build a ZMQ Server-Client architecture.
7. Closed-Loop Testing: Run control on real hardware and monitor for jittering/stop-and-go behavior.
8. Optimization: Tune RTC parameters and trajectory smoothing.

Trajectory Filtering

1. Script Filtering: Remove samples with excessive latency in a single camera stream, excessive timestamp differences between camera streams, or abnormal jumps in robot state sequences.
2. Manual Filtering: Replay trajectories with synchronized visualization to remove blurry frames, failed task executions, poor synchronization, or low-quality/discontinuous trajectories.

Trajectory Preprocessing

• Timestamp alignment: Align camera frames and robot joint states to a shared time base.
• Head-tail trimming: Remove idle segments at the start and end.
• Splitting: Split long trajectories (several minutes) into multiple subtasks.

Format Conversion

• Convert all data to the LeRobot format using provided conversion scripts for GR00T compatibility.

Choose hardware based on your required inference frequency (replanning rate). Note that a ~10 Hz inference rate can sustain ~30 FPS execution via action chunking and asynchronous inference.

Minimum Requirements:

• 1 GPU with 16 GB+ VRAM
• CUDA 12.6+

Performance Profiles:

• High-frequency (30+ Hz): Best for reactive, low-latency closed-loop control. Requires H100 or RTX Pro 6000 with the TensorRT Full Pipeline.
• Recommended Minimum (10+ Hz): Sufficient for typical manipulation tasks. Achievable with AGX Thor, DGX Spark (using TensorRT), or most dGPUs using torch.compile.
• Low-frequency (< 5 Hz): Suitable only for slow, non-reactive tasks. This applies to Orin platforms where TensorRT 10.3 only supports DiT-only mode and cannot accelerate the backbone.

Install development dependencies and run the pytest suite. Use targeted test paths for faster local checks. Note that GPU-marked tests require CUDA hardware.

uv sync --python 3.12 --extra dev
uv run python -m pytest

Evaluate a fine-tuned model's performance using gr00t/eval/open_loop_eval.py. This compares predicted actions against ground truth trajectories.

Example command:

uv run python gr00t/eval/open_loop_eval.py \
    --dataset-path ./demo_data/cube_to_bowl_5 \
    --embodiment-tag NEW_EMBODIMENT \
    --model-path /tmp/so100/checkpoint-2000 \
    --traj-ids 0 \
    --execution-horizon 16 \
    --steps 400 \
    --modality-keys single_arm gripper

If required during setup, you can install a specific version of lerobot from GitHub while skipping Git LFS smudge to save time/bandwidth during the installation process.

GIT_LFS_SKIP_SMUDGE=1 uv pip install "lerobot @ git+https://github.com/huggingface/lerobot.git@c75455a6de5c818fa1bb69fb2d92423e86c70475"

Orin uses CUDA 12.6 and Python 3.10 (JetPack 6.2).

Important Limitation: On Orin, only the DiT (action head) TensorRT export is supported. Use --export-mode dit_only.

Docker (Recommended): Build the container from the repo root:

cd docker && bash build.sh --profile=orin && cd ..

Then use the Shared Docker Workflow with image gr00t-orin.

Bare Metal:

1. Run the one-time installation script:

bash scripts/deployment/orin/install_deps.sh

2. In every new shell, activate the environment:

source .venv/bin/activate
source scripts/activate_orin.sh

Storage Tip: If eMMC space is low, redirect the HuggingFace cache to an NVMe SSD: export HF_HOME=/path/to/ssd/.cache/huggingface.

Build the standard GR00T Docker image from the repository root. This image uses nvidia/cuda:12.8.0-devel-ubuntu24.04 and installs dependencies into /opt/gr00t-venv.

Note: The image does not include a source checkout. You must clone or pull the repository inside the container after starting it.

bash docker/build.sh

Use the --profile flag with docker/build.sh to build specialized images for specific hardware architectures:

• Thor Container (Jetson Thor / CUDA 13): bash docker/build.sh --profile=thor
• Spark Container (DGX Spark / CUDA 13): bash docker/build.sh --profile=spark
• Orin Container (Jetson Orin / CUDA 12.6): bash docker/build.sh --profile=orin