NeMo Evaluator

Security

• Egress: The ECS task security group allows all egress to support ECR pulls, S3, and DockerHub.
• Ingress: Inbound traffic is restricted to ssh_tunnel_sshd_port and only from the CIDRs specified in orchestrator_allowed_cidrs. The module explicitly rejects 0.0.0.0/0 for security.

Lifecycle and Rotation

• S3 Buckets: Uses bucket_prefix for unique global naming. force_destroy = true is enabled to allow terraform destroy to clean up even if objects exist.
• Secrets: recovery_window_in_days = 0 is used so secrets are deleted immediately upon destruction.
• Docker Hub Credentials: The secret_string for Docker Hub is wrapped in lifecycle { ignore_changes } to prevent subsequent applies from overwriting manual rotations. To rotate credentials via Terraform, use: terraform apply -replace=...aws_secretsmanager_secret_version.dockerhub

NeMo Evaluator (NEL) supports resolving benchmarks from external sources using specific URI schemes and namespace prefixes. This allows you to run benchmarks from lm-evaluation-harness, NeMo Skills, VLMEvalKit, or remote servers.

| Syntax | Source | Example |
|--------|--------|---------|
| `nel eval run --bench <name>` | Built-in registry | `nel eval run --bench mmlu` |
| `nel eval run --bench lm-eval://<task>` | lm-evaluation-harness | `nel eval run --bench lm-eval://aime25` |
| `nel eval run --bench skills://<name>` | NeMo Skills | `nel eval run --bench skills://mmlu-pro` |
| `nel eval run --bench vlmevalkit://<dataset>` | VLMEvalKit | `nel eval run --bench vlmevalkit://MMBench_DEV_EN` |
| `nel eval run --bench gym://<host:port>` | Remote Gym server | `nel eval run --bench gym://localhost:9090` |
| `nel eval run --bench harbor://<suite>@<version>` | Harbor task registry | `nel eval run --bench harbor://swebench-verified@1.0` |
| `nel eval run --bench container://<image>#<task>` | Legacy container | `nel eval run --bench container://nvcr.io/image#task` |

NeMo Evaluator provides several core capabilities for LLM evaluation:

• Unified Environment Registry: Everything (benchmarks, skills, gym remotes, etc.) resolves through a single registry.
• Rapid Benchmark Definition: Use @benchmark and @scorer decorators to define complete benchmarks in under 10 lines of Python.
• Pluggable Solvers: Swap inference strategies per benchmark via configuration using solvers like simple, harbor, tool_calling, gym_delegation, or openclaw.
• Flexible Execution Backends: Run evaluations locally, in Docker, or on SLURM clusters with automatic model deployment.
• Resilience: Features per-benchmark checkpointing and failure isolation. You can resume partially completed suites using the --resume flag.
• Statistical Analysis: Compare runs using McNemar's exact test, paired flip analysis, and confidence intervals.
• Quality Gates: Gate releases using per-benchmark policy thresholds.
• Built-in Benchmarks: Includes 17 benchmarks such as MMLU, MMLU-Pro, MATH-500, GPQA, GSM8K, HumanEval, and more.

The tool_calling solver allows NeMo Evaluator (NEL) to drive the agentic loop itself, providing full observability into model calls, tool calls, and conversation turns. This is an alternative to delegated solvers like gym_delegation or harbor, which are opaque to NEL.

Using Gym HTTP Tools

To use a Gym Resource Server, set resource_service to the service providing the tools. NEL discovers tools via /openapi.json and dispatches calls via POST /{tool_name}.

Using Sandbox Tools (NEL as the Agent)

By setting sandbox_tools: true, NEL provides a canonical set of tools that execute inside a configured sandbox (e.g., Docker). Available tools include:

• bash: Execute shell commands.
• file_read: Read file contents.
• file_write: Write file contents.
• str_replace_editor: View or make targeted replacements in files.

Key Configuration Options

• sandbox_tools: true: Enables the four sandbox tools. Requires a sandbox config.
• response_mode: sandbox_artifact: The answer is the sandbox state (e.g., a git patch) rather than the last model message. Useful for SWE-bench.
• max_turns: Maximum model call rounds. If reached, the solver returns error: "max_turns_exhausted".
• tool_timeout: Per-tool-call timeout in seconds (default: 180).
• max_output_chars: Truncation limit for tool output fed back to the model (default: 16384).

benchmarks:
  - name: swebench-multilingual
    solver:
      type: tool_calling
      service: model
      resource_service: tools
      max_turns: 50

NeMo Evaluator (NEL) uses Hydra-style config composition to allow for reusable fragments, base-config inheritance, and self-referencing interpolation. This is opt-in via the defaults: key at the top of your YAML configuration.

Core Concepts

• defaults: list: Defines which config fragments or base configs to load and merge.
• _base_: Used within a defaults: list to inherit from an existing complete configuration. Inheritance can be chained (e.g., Grandparent $\rightarrow$ Parent $\rightarrow$ Child).
• _self_: A special entry in the defaults: list that determines where the current config's body is placed in the merge order. If omitted, it defaults to the last position, giving the current config the highest priority.
• Config Groups: Fragments are organized into directories that map to top-level keys in the final configuration. For example, a fragment at services/nemotron_fp8.yaml will be merged into the services: key.

defaults:
  - _base_: 24_slurm_swebench_verified_super

benchmarks:
  - name: harbor://swebench-verified@1.0
    max_concurrent: 30
    solver:
      agent: mini-swe-agent
      agent_kwargs: { max_iterations: 30 }
    sandbox: { concurrency: 30 }

output:
  dir: ./eval_results/swebench-verified-miniswe