DOCSEARCHbyAlgolia
Submit Docs

NVIDIA Nemotron

repository·main·Indexed 23 days ago

https://github.com/nvidia-nemo/nemotron

A family of open, high-efficiency multimodal models for agentic AI. This repository provides reproducible training recipes, modular building blocks, and deployment guides for models ranging from edge-optimized Nano to datacenter-scale Ultra. It includes a two-layer architecture separating experiment orchestration (Execution Layer) from training logic (Runtime Layer), as well as a complete pipeline for embedding model fine-tuning across six stages from synthetic data generation to deployment.

Tokens
555.7K
Snippets
979
Records
2.4K
Agent score
77%

What's inside Nemotron

  1. Overview of Nemotron 3 Ultra DAPO Training with NeMo Gym

    main

    This workflow uses NeMo RL for policy training and NeMo Gym for rollout and reward orchestration to perform DAPO/GRPO reinforcement learning on the nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 model.

    NeMo Gym interacts with vLLM via an OpenAI-compatible HTTP server and performs verification (e.g., math_with_judge) over generated responses.

    Default Profile Specifications:

    • Compute: 18 nodes total (16 for Megatron policy training, 2 for non-colocated vLLM generation).
    • GPU Configuration: 4 GPUs per node.
    • Parallelism: Megatron TP=8, CP=8, EP=64; vLLM TP=8.
    • Sequence Length: 4k total; 2k-token rollout budget.
    • Batching: 64 prompts x 16 generations per step.
    • Training: 100 steps with checkpoints every 10 steps.

  2. Overview of Nemotron 3 Nano 30B-A3B

    main

    Nemotron 3 Nano is an open hybrid Mamba-Transformer Mixture-of-Experts (MoE) model designed for reasoning, chat, and agentic tasks. It is optimized for high efficiency and throughput, serving as a successor to Nemotron 2 Nano.

    Key Specifications

    • Architecture: Hybrid Mamba-Transformer with sparse MoE.
    • Parameters: 31.6B total parameters; 3.2B active parameters per forward pass (3.6B including embeddings).
    • Context Window: Supports up to 1M tokens.
    • Pretraining: 25T tokens (including 3T+ new unique tokens over Nemotron 2).
    • Training Pipeline: Pretraining $\rightarrow$ SFT (Supervised Fine-Tuning) $\rightarrow$ RLVR (Reinforcement Learning from Verifiable Rewards) + RLHF (Reinforcement Learning from Human Feedback).

    Performance Highlights

    NVIDIA reports significant throughput advantages in generation-heavy settings (8K input / 16K output) on a single H200 GPU:

    • Up to 3.3x higher throughput than Qwen3-30B-A3B-Thinking-2507.
    • Approximately 2.2x higher throughput than GPT-OSS-20B.

  3. Overview of Nemotron 3 Ultra Model Specifications

    main

    Nemotron 3 Ultra is a hybrid Mamba-Attention LatentMoE model with the following core specifications:

    • Architecture: 550B total parameters / 55B active parameters.
    • Context Window: 1M context length.
    • Pretraining: Trained on 20T text tokens using a WSD (Warmup-Stable-Decay) schedule and an NVFP4 recipe.
    • SFT (Supervised Fine-Tuning): 204,800 packed samples covering domains such as long-context, safety, search, terminal/tool use, SWE, math/proof, code, CUDA, RTL, multilingual, and chat.
    • MOPD: An RLVR (Reinforcement Learning from Verifiable Rewards) student distilled from over ten specialized teachers via asynchronous dense token-level on-policy guidance.
    • Reasoning Modes: Supports reasoning-off, regular, and medium-effort modes. MTP (Multi-Token Prediction) Boosting is used to improve speculative-drafter acceptance.
    • Quantization/Inference: Features a 5.03 BPE NVFP4 operating point and includes one W4A4/W4A16 checkpoint.

  4. Overview of Nemotron Data Prep steps

    main

    Nemotron Data Prep transforms raw or semi-structured training data into trainer-ready artifacts. The specific step you choose depends on your downstream trainer's requirements.

    NeedStepProduces
    Pack chat JSONL for Megatron-Bridge SFT or PEFTdata_prep/sft_packingpacked_parquet
    Tokenize text into Megatron pretraining shardsdata_prep/pretrain_prepbinidx + blend.json
    Resolve and shard RL prompt or preference datadata_prep/rl_preptraining_jsonl (sharded)

  5. Overview of Nemotron 3 Nano Omni RL sub-stages

    main

    The Stage 1 Omni RL process consists of three distinct RL flows. Each flow uses a specific algorithm, source dataset, and produces a unique model artifact.

    Sub-stageAlgorithmSource datasetOutput artifact
    stage1_mpo/Mixed Preference OptimizationOpenGVLab/MMPRomni3-rl-mpo-model:latest
    stage2_text_rl/GRPO (text-only)nvidia/Nemotron-3-Nano-RL-Training-Blendomni3-rl-text-model:latest
    stage3_vision_rl/GRPO (vision)OpenGVLab/MMPR-Tinyomni3-rl-vision-model:latest

    Note: This recipe reproduces 3 of the 25 environments used in the full upstream alignment. The remaining 22 environments use internal or third-party data and are not included here.

  6. Overview of Nemotron 3 Super (120B)

    main

    Nemotron 3 Super is NVIDIA's flagship open model in the Nemotron 3 family, specifically designed for high-throughput agentic reasoning. It uses a hybrid architecture combining Mamba-2 sequence modeling, sparse MoE capacity (LatentMoE), attention anchor layers, and shared-weight Multi-Token Prediction (MTP) heads.

    Key Specifications:

    • Model Name: Nemotron 3 Super 120B-A12B
    • Parameters: 120.6B total / 12.7B active per forward pass
    • Architecture: Hybrid Mamba-Transformer LatentMoE
    • Context Length: Up to 1,048,576 tokens
    • MTP: 2 shared-weight MTP layers
    • Pretraining: 25T tokens
    • Quantization Variants: FP8 (for Hopper) and NVFP4 (for Blackwell)

  7. Overview of Nemotron Training Recipes

    main

    The Nemotron repository provides complete, reproducible training pipelines that cover the entire lifecycle from raw data to deployment-ready models. These pipelines are designed to show how different stages (Pretraining, SFT, RL, etc.) interact and how to balance competing objectives.

    Each recipe includes:

    • Synthetic Data Generation: Using NVIDIA-NeMo/DataDesigner.
    • Data Curation: Using NVIDIA NeMo Curator for filtering and quality enhancement.
    • Training: Complete loops using NVIDIA-NeMo/Megatron-Bridge (Megatron models), NVIDIA-NeMo/Automodel (HuggingFace models), or NVIDIA-NeMo/NeMo-RL (Reinforcement Learning).
    • Evaluation: Benchmarking via NVIDIA NeMo Evaluator.
    • Documentation: Detailed stage-by-stage explanations.

  8. Overview of Nemotron Development Patterns

    main

    The Nemotron repository provides a collection of architectural and procedural patterns designed to guide developers through complex LLM workflows. These patterns address specific challenges in model training, data preparation, evaluation, and deployment.

    Patterns are categorized by their application area, such as:

    • Evaluation: Designing custom benchmarks (BYOB), evaluating before/after training, and validating RL rewards.
    • Data Preparation: SFT sequence packing, data blending for CPT/SFT, and handling tokenizer-locked data.
    • Training & Fine-tuning: Choosing between LoRA and full fine-tuning based on dataset size, and scoping CPT data blends.
    • Conversion & Deployment: Checkpoint safety during format conversion (HuggingFace to Megatron) and exporting to TensorRT-LLM for production.
    • Translation & Multilingualism: Strategies for translating corpora, checking tokenizer coverage, and choosing between LLM-based or NMT-based translation.

  9. Overview of the Nemotron 3 Omni Training Pipeline

    main

    The Nemotron 3 Omni training pipeline is a multi-stage process consisting of data preparation, model importing, Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL).

    Pipeline Flow

    1. Data Preparation: Synthetic document data is processed via omni3 data prep sft.
    2. Model Import: A Hugging Face checkpoint is imported using omni3 model import pretrain.
    3. SFT (Stage 0): Supervised Fine-Tuning is performed using omni3 sft.
    4. RL (Stage 1): Reinforcement Learning is executed through three sub-stages:
      • MPO: Multimodal preference optimization.
      • Text RL: GRPO-based alignment on the nvidia/Nemotron-3-Nano-RL-Training-Blend dataset.
      • Vision RL: GRPO on the MMPR-Tiny dataset.

    Repository Structure

    • src/nemotron/recipes/data/curation/: Filtering and curating existing corpora.
    • src/nemotron/recipes/data/sdg/: Generating new datasets (e.g., the long-document SDG pipeline).
    • src/nemotron/recipes/omni3/: Family-specific training, RL, and evaluation stages.

  10. Overview of the Nemotron 3 Ultra Training Recipe

    main

    The Nemotron 3 Ultra Training Recipe provides a complete data-preparation and training pipeline for Nemotron 3 Ultra 550B-A55B. This model is a hybrid Mamba-Transformer MoE (Mamba-2 + Attention + LatentMoE) featuring multi-token prediction (MTP) and support for up to 1M tokens of context.

    The recipe is divided into two primary stages:

    1. Stage 0: Pretraining: Uses the Megatron-Bridge framework to train on an open pretrain mixture, resulting in a Base Model checkpoint.
    2. Stage 1: SFT (Supervised Fine-Tuning): Uses the Megatron-Bridge framework for instruction and agentic tuning, resulting in an Instruct Model.

    Note: While the full Ultra program includes RLVR, MOPD, and MTP Boosting, only the Pretraining and SFT stages are publicly available in this recipe.

  11. Overview of Nemotron 3 Omni Training Recipe

    main

    The Nemotron 3 Omni Training Recipe is a multimodal post-training pipeline for Nemotron 3 Nano Omni, a 30B-A3B hybrid mixture-of-experts (MoE) model. The model supports text, image, video, and audio modalities with a maximum context length of 262K tokens.

    Unlike previous versions, Omni starts from a General Availability (GA) checkpoint and manages its own stage-local container builds. The pipeline is divided into two main stages:

    1. Stage 0: SFT (Supervised Fine-Tuning): Fine-tunes the GA checkpoint on multimodal datasets like Valor32k.
    2. Stage 1: RL (Reinforcement Learning): A multi-stage process consisting of MPO (Multi-objective Preference Optimization), followed by text-specific RL, and finally vision-specific RL.

  12. Overview of the Nemotron Data Preparation Module

    main

    The nemotron.data_prep module is responsible for last-mile data processing: transforming curated datasets into training-ready formats. It is designed to sit between data curation (e.g., using NeMo Curator) and model training (e.g., Megatron-Bridge or NeMo-RL).

    Key features include:

    • Distributed processing: Built on Ray to scale from single machines to clusters.
    • Cloud-native I/O: Supports hf:// (HuggingFace), s3:// (S3), gs:// (GCS), and local paths via fsspec.
    • Deterministic & Resumable: Uses frozen shard plans for reproducibility and allows skipping completed shards on restart.
    • Core Tasks: Tokenization, chat templating, sequence packing, and loss mask generation.