repository·main·Indexed 23 days ago
https://github.com/nvidia-nemo/rlA scalable and efficient post-training library for models ranging from 1 GPU to 1000s, and from tiny to >100B parameters. Includes nrl-k8s, a config-driven launcher for running NeMo-RL recipes on Kubernetes clusters using the KubeRay operator to automate RayCluster deployment, daemon management, and training job execution.
The nemo-rl-session-memory skill provides coding agents with durable working-session memory. It allows agents to preserve and recover context across various interruptions, such as:
Important state is written periodically to the repository's session directory to ensure continuity.
The NeMo RL project provides a Ray-based infrastructure for distributed Reinforcement Learning (RL) training. Key capabilities include:
Group Relative Policy Optimization (GRPO) in NeMo RL is an implementation designed for reinforcement learning from human feedback (RLHF) or similar processes. The implementation covers four primary pillars:
For specific configuration details, refer to the examples/configs/grpo_math_1B.yaml configuration file.
NeMo RL is an open-source post-training library within the NeMo Framework. It is designed to streamline and scale reinforcement learning (RL) methods for multimodal models, including Large Language Models (LLMs) and Vision-Language Models (VLMs).
Key features include:
NeMo RL provides model-family specific guidance for post-training tasks. This guidance covers recipe selection, recommended generation settings, and known issues.
Key Principles:
examples/configs/recipes/ are the definitive source of truth for configuration.model-support.md documentation.NeMo RL provides support for various training algorithms used in the post-training of large language models. These algorithms are categorized by their deployment capability on Single Node or Multi-node configurations.
model-support.md documentation.Cross-Tokenizer (X-Token) distillation allows NeMo RL to perform off-policy distillation between a student model and a teacher model that use different tokenizers (e.g., distilling a Qwen3-4B teacher into a Llama-3.2-1B student).
Because the vocabularies only partially overlap, the system uses a projection matrix to bridge the gap. This matrix is a sparse [V_student, top_k] tensor that maps each student token to the teacher tokens it most plausibly corresponds to. During training, the student's logits are multiplied by this matrix to project them into the teacher's vocabulary space, allowing for a direct comparison of probability distributions.
Multi-Teacher On-Policy Distillation (MOPD) is a training paradigm that distills one or more teacher models into a student policy. It replaces the standard GRPO reward-based advantage with a token-level distillation advantage.
Key characteristics:
adv_estimator: opd) and utilizes NeMo Gym for rollouts.Dual-Clip Asymmetric Policy Optimization (DAPO) is an extension of GRPO that introduces asymmetric clipping. Unlike standard GRPO or PPO, which use identical values for minimum and maximum clipping, DAPO allows for distinct ratio_clip_min and ratio_clip_max parameters. This provides more fine-grained control over policy updates by setting asymmetric bounds on the probability ratio.
Technically, DAPO is implemented using the same ClippedPGLossFn as GRPO, making it fully compatible with existing GRPO training infrastructure and configurations.
/tokenize endpoint. This ensures that the tokenization used during training exactly matches the model and tokenizer used during generation.NeMo RL includes advanced capabilities for optimizing large-scale model training: