NVIDIA NeMo Agent Toolkit

The Dynamo Monitoring Stack provides a Prometheus and Grafana setup for monitoring the Dynamo LLM inference stack with a Thompson Sampling router. It collects metrics at a 2-second resolution directly from the ai-dynamo Prometheus API to provide per-request granularity.

Note: This integration is EXPERIMENTAL. It requires Dynamo >= 1.1.0. In these versions, dynamo.sglang normalizes request priority (higher values = higher priority) and rejects the --schedule-low-priority-values-first flag.

The Profiler Module provides profiling and forecasting for workflows. It instruments workflow execution to:

• Collect real-time usage statistics via callbacks.
• Record per-invocation statistics (e.g., tokens used, time between calls, LLM calls).
• Store data for offline analysis.
• Forecast usage metrics (e.g., tokens, tool usage) using time-series models like Linear or Random Forest.
• Compute performance metrics such as latency, throughput, bottlenecks, and concurrency spikes.

This allows developers to stress test workflows in pre-production and receive sizing guidance based on observed performance.

The nvidia_nat_config_optimizer subpackage is designed for workflow configuration and prompt optimization within the NeMo Agent Toolkit. It provides specialized components located under the nat.plugins.config_optimizer namespace.

Key capabilities include:

• GA Prompt Optimization: Using Genetic Algorithms (GA) to optimize prompts.
• Optuna-based Parameter Optimization: Utilizing the Optuna framework to optimize workflow parameters.

The Agno Personal Finance workflow is a beginner-level implementation that integrates the Agno multimodal agent library with the NVIDIA NeMo Agent Toolkit.

Key Capabilities:

• Personalized Planning: Generates tailored budgets, investment strategies, and savings goals using NVIDIA NIM models.
• Automated Research: Uses the SERP API to perform real-time market research and gather financial data.
• Multi-Framework Architecture: Combines Agno's model-agnostic agent capabilities with NeMo Agent Toolkit's workflow management and tool integration system.

The Automated Description Generation Workflow is a beginner-level workflow designed to enhance RAG (Retrieval-Augmented Generation) services. It builds upon the RAG service to automatically generate intelligent descriptions for collections within a Vector Database (currently focused on Milvus).

These descriptions are used by the multi-query collection tool to improve retrieval accuracy across multiple collections by providing better context for document retrieval. The workflow uses a map-reduce summarization approach: it retrieves documents using dummy embeddings, generates local summaries via an LLM, and then uses map-reduce techniques to produce a final collection description.

The NeMo Agent Toolkit provides a finetuning harness for in-situ reinforcement learning of agentic LLM workflows. This allows you to iteratively improve agents by training them on the same workflows used in production, enabling them to learn from interactions with environments, tools, and users.

Core Principles

• Decoupled Architecture: Training logic is separated from backends, supporting various RL frameworks (e.g., OpenPipe ART, NeMo Aligner).
• In-Situ Training: Agents are trained within their production-like workflow environment.
• Flexible Targeting: You can finetune specific functions or entire workflows.
• Composable Components: The system uses three pluggable components: TrajectoryBuilder, TrainerAdapter, and Trainer.

Dynamo is NVIDIA's high-performance LLM serving platform featuring KV cache optimization. The NeMo Agent Toolkit integration provides two main capabilities:

1. Dynamo LLM Support: Implementation of DynamoLLM (located in packages/nvidia_nat_core/src/nat/llm/dynamo_llm.py) to enable NeMo Agent Toolkit inference on Dynamo runtimes.
2. Scalable Startup Scripts: A set of scripts designed for NVIDIA Hopper and Blackwell GPU servers to run NeMo Agent Toolkit runtimes at scale.

This beginner-level example demonstrates a Retrieval-Augmented Generation (RAG) agent designed to answer questions about LangSmith documentation.

Core Components:

• Webpage Query Tool: A tool that scrapes documentation from https://docs.smith.langchain.com and uses vector search to retrieve information.
• ReAct Agent: A react_agent that uses reasoning to decide when to call the web query tool.
• Document Retrieval & Embedding: Automatically generates embeddings from scraped web content for semantic search.
• Configuration: The entire workflow (scraping, embeddings, and agent reasoning) is managed via a YAML-based configuration file.

The nvidia-nat-config-optimizer is a tool for workflow configuration and prompt optimization within the NVIDIA NeMo Agent Toolkit. It uses genetic algorithms and numeric optimizers (via Optuna) to optimize configuration-level values, such as hyperparameters and prompts.

Note: This tool is scoped strictly to configuration-level optimization; it does not handle runtime or inference optimizations.