AgentLab provides several mechanisms to track and verify the reproducibility of agent evaluations on dynamic benchmarks:

• Metadata Tracking: The Study class automatically includes a dictionary of reproducibility information, such as benchmark versions, package versions, and commit hashes.
• Automatic Uploads: When installed via pip install -e . from a cloned repository, the Study class can automatically upload results to reproducibility_journal.csv.
• ReproducibilityAgent: A specialized agent that can be run on an existing study. It attempts to re-run the same actions on the same task seeds. It provides a visual diff of the two prompts in the AgentInfo HTML tab of AgentXray, allowing you to inspect changes between executions (Note: this is a beta feature).
• Leaderboard Support: Reproducible agents can have their results highlighted in the leaderboard via a special column containing reproduction information.

AgentLab requires Python 3.11 or 3.12. Follow these steps to set up your environment:

1. Install the package via pip:

   pip install agentlab

2. Install Playwright (required for browser automation):

   playwright install

3. Prepare Benchmarks: Follow the specific setup instructions for your chosen benchmark (e.g., WebArena, WorkArena) as listed in the supported benchmarks table.
4. Configure Environment Variables:
   • AGENTLAB_EXP_ROOT: The root directory for experiment results (defaults to $HOME/agentlab_results).
   • OPENAI_API_KEY: Required if using OpenAI models.
   • OPENROUTER_API_KEY: Required if using OpenRouter models.
   • AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT: Required if using Azure OpenAI models.

pip install agentlab
playwright install
export AGENTLAB_EXP_ROOT=/path/to/results
export OPENAI_API_KEY=sk-... 

Launching experiments with Open Source Software (OSS) LLMs is a manual multi-step process. Follow these steps to set up and run your experiments:

1. Launch the LLM(s):
   • Open llm_configs.py.
   • In the if __name__ == "__main__": block, set the model variable to a model name found in CHAT_MODEL_ARGS_DICT.
   • Execute llm_configs.py.
2. Retrieve the URL:
   • After launching, the eai CLI will output a URL in the terminal. You can also retrieve it by running:

     eai job ls -r

3. **Configure the Model URL**:
   - In `llm_configs.py`, find the `CHAT_MODEL_ARGS_DICT`.
   - Set the `model_url` key for your chosen OSS LLM(s) to the URL retrieved in step 2.
4. **Configure the Experiment**:
   - In `exp_configs_OSS.py`, set the `model_name_list` to include the names of the OSS LLMs you just launched.
5. **Launch the Experiment**:
   - Run your preferred experiment configuration within `exp_config_OSS.py`.

Step 2: Retrieve URL

eai job ls -r

Step 3: Update CHAT_MODEL_ARGS_DICT in llm_configs.py

CHAT_MODEL_ARGS_DICT = { "your-model-name": { "model_url": "URL_FROM_TERMINAL", # ... other args } }

Experiments are managed using the make_study function. You define a study by specifying a benchmark and a list of agent configurations (extending bgym.AgentArgs).

Running a new study

from agentlab.agents.generic_agent import AGENT_4o_MINI 
from agentlab.experiments.study import make_study

study = make_study(
    benchmark="miniwob",  # e.g., "webarena", "workarena_l1", etc.
    agent_args=[AGENT_4o_MINI],
    comment="My first study",
)

study.run(n_jobs=5)

Relaunching incomplete or errored tasks

If a study was interrupted, you can load it from the directory and run only the failed or unfinished tasks:

from agentlab.experiments.study import Study
study = Study.load("/path/to/your/study/dir")
study.find_incomplete(include_errors=True)
study.run()

Best Practices

• Parallelism: You can typically run 10–50 jobs in parallel on a single computer depending on RAM. AgentLab uses ray for large-scale parallel execution.
• Debugging: To debug using a tool like VSCode, run the experiment with n_jobs=1 to allow for breakpoints.
• WebArena Dependencies: For (Visual)WebArena, AgentLab accounts for task dependencies in the Ray backend to prevent state corruption. For smoother parallel performance, consider benchmarks like WorkArena.