Harbor Framework

The MLGym-Bench adapter evaluates language model agents on machine learning tasks including computer vision, reinforcement learning, tabular ML, and game theory. It repackages exercises from the MLGym repository, requiring agents to write Python code to train and evaluate models.

Key Characteristics:

• Metrics: Uses continuous metrics (accuracy, RMSE, reward, etc.) by default. Use --no-continuous-metrics for binary pass/fail evaluation.
• Tasks: Surfaces 11 tasks (e.g., titanic, imageClassificationCifar10, rlMountainCarContinuous).
• Environment: Provides Dockerized environments with PyTorch and CUDA support. Use --no-cuda for CPU-only mode.
• Computation: Tasks are computationally intensive (training from scratch). Sequential execution (n_concurrent_trials: 1) is strongly recommended.

The RefAV (Referring Expression Grounding for Autonomous Vehicle Scenario Mining) adapter allows Harbor to evaluate agents on their ability to identify driving scenarios from natural language descriptions using AV2 sensor log data. Agents must generate Python code that identifies specific (track_uuid, timestamp_ns) pairs.

Key Details:

• Total Tasks: 1,500 tasks (150 AV2 logs with ~10 prompts each).
• Parity Subset: A deterministic 50-task subset (seed=42) for faster testing.
• Evaluation Metric: HOTA-Temporal (Higher Order Tracking Accuracy).
• Environment: Dockerized with CPU-only PyTorch, av2-api-sm, and TrackEval (~3.6GB image).
• Agent Support: Standard Harbor CLI agents like codex and claude-code work out of the box.

Harbor is a framework designed for evaluating and optimizing agents and models within containerized environments. It provides modular interfaces for managing environments, agents, and tasks at scale.

Key features include:

• Modular interfaces for environments, agents, and tasks.
• Pre-integrated support for popular CLI agents.
• A registry of popular benchmarks and datasets.
• Horizontal scaling via integrations with cloud sandbox providers (e.g., Daytona, Modal, E2B, Runloop, Tensorlake, LangSmith, Blaxel, Novita Sandbox, EC2, and Beam).
• Optimization capabilities through integrations with frameworks like SkyRL and GEPA.

The KUMO adapter converts KUMO (procedurally generated reasoning games) into Harbor-compatible tasks for evaluating interactive, multi-step reasoning.

Key Details:

• Task Type: Interactive reasoning games where agents query actions, observe outcomes, and identify a hidden truth.
• Domains: 100+ domains (e.g., medicine, education, music, algorithms).
• Runtime: Python-based (agent CLI + verifier service) running in Docker environments.
• Task Count: Supports 5,300 tasks (5,050 easy, 250 hard) plus an additional 99,950 MedicalOOD instances.
• Evaluation: Exact-match accuracy on the final truth written to /app/answer.txt.
• Security: Secrets are isolated; environment/secret.json is only available to the verifier image, while the agent only receives knowledge_book.txt.

The PIXIU adapter supports 29 subcategories across 8 categories. It automatically fetches data from Hugging Face and generates oracle solutions.

Supported Categories & Metrics:

• Classification: Accuracy, F1, Macro F1, MCC, Missing.
• TSA (Targeted Sentiment Analysis): RMSE, Missing.
• QA: Accuracy, Missing.
• NER / Sequential Labeling: Entity-level F1, token-level F1.
• Relation Extraction: Precision, Recall, F1.
• Summarization (Extractive/Abstractive): ROUGE-1/2/L, BERTScore, BARTScore.

The GAIA2 adapter packages Meta's GAIA2 validation scenarios into Harbor task directories. It supports 800 public validation scenarios across five configurations: execution, search, adaptability, time, and ambiguity.

There are two primary packaging modes:

1. ARE native mode (gaia2): Runs the official ARE oracle path or the official ARE default agent path for parity experiments.
2. CLI mode (gaia2-cli): Runs a standard CLI agent in a main container and exposes the ARE environment via an MCP sidecar.

Parity experiments are performed on a deterministic 100-task stratified subset (20 tasks per config) using a fixed seed.

The CRMArena adapter supports two agent types with different interaction patterns:

1. CLI Agents (e.g., claude-code, openhands, aider):

   • Prompt: Receives the full instruction.md.
   • Schema: Uses a simplified table in instruction.md containing key fields.
   • Salesforce Access: Writes Python code using simple_salesforce.
   • Answer Output: Writes Python code to /app/answer.txt.

2. Custom ReAct Agent:

   • Prompt: Selectively parses instruction.md (only ## Question, ## Task-Specific Instructions, and ## Additional Context).
   • Schema: Uses a detailed _SCHEMA_STRING in the system prompt (all fields, types, and references).
   • Salesforce Access: Emits <execute>SOQL/SOSL</execute> tags in a ReAct loop.
   • Answer Output: Emits <respond>answer</respond> tags, which are written to /app/answer.txt.

When using the ScienceAgentBench adapter, keep these requirements and limitations in mind:

• Data Availability: SAB datasets, gold programs, and evaluation scripts are not public; they must be downloaded separately from SAB's SharePoint. Only HuggingFace annotations are publicly available.
• Resource Intensity: Tasks such as deepchem, scvi, and DeepPurpose require at least 16GB of memory and can take over 30 minutes to complete.
• API Costs: Category B evaluations involve 3 GPT-4o vision API calls per task.
• Non-deterministic Tasks: Tasks sab_72 and sab_102 are inherently stochastic due to ML training without fixed seeds.

The Harbor adapter provides two ways to run BixBench tasks:

1. BixbenchJupyterAgent: A custom agent designed to mimic the original benchmark's Jupyter notebook environment for fair parity comparison.
2. Terminal Agents: Generic agents (e.g., codex, claude code) used via the bixbench-cli dataset. These agents use different prompts than the Jupyter version because they do not rely on specific Jupyter tools like submit_answer or edit_cell.