DOCSEARCHbyAlgolia
Submit Docs

MLAgentBench

repository·main·Indexed 43 days ago

https://github.com/snap-stanford/mlagentbench

A benchmarking suite designed to evaluate AI agents on their ability to perform end-to-end machine learning experimentation tasks, including data processing, model development, and result analysis. It supports various LLM providers such as OpenAI, Claude, CRFM, Gemini Pro, and HuggingFace, and provides tools for task preparation, execution in Docker sandboxes, and performance evaluation.

Tokens
1.7K
Snippets
7
Records
10
Agent score
6%

What's inside MLAgentBench

  1. Understand the log directory structure

    main

    When running a task, the specified --log-dir will contain the following structure:

    • agent_log/
      • main_log: The primary log showing the agent's research process.
      • agent_*.json: Saved agent states.
    • env_log/
      • tool_logs/: Logs from tools used by the agent.
      • traces/: Snapshots of the agent's workspace.
      • trace.json: The interaction trace of the agent.
      • overall_time.txt: Total execution time.
      • error.txt: Generated if a system error occurs.

  2. Prepare task datasets

    main

    Datasets are prepared automatically when a task is run for the first time. However, you can pre-prepare them using the MLAgentBench.prepare_task module.

    Note for Kaggle datasets: You must have your Kaggle API credentials (~/.kaggle/kaggle.json) configured. If using Docker, ensure .kaggle/kaggle.json is in the MLAgentBench root folder and run:

    export KAGGLE_CONFIG_DIR=/MLAgentBench/.kaggle
pip install kaggle
sudo apt-get install unzip
    python -u -m MLAgentBench.prepare_task <task_name> $(which python)

  3. Systematic Benchmark Workflow

    main

    To conduct a full benchmark evaluation, follow these steps:

    1. Run Experiments: Use run_experiments.sh to run parallel experiments across different tasks and agents. This generates logs in final_exp_logs/<model_name>/<run_timestamp>/.
    2. Run Baselines: Execute baseline.sh on all tasks to establish baseline performance.
    3. Generate Evaluations: Run eval.sh with the appropriate models and tasks to produce evaluation JSONs (including baselines).
    4. Analyze Results: Use plot.py in the MLAgentBench directory to generate plots from the generated JSONs. Note: You may need to manually update paths and names in plot.py where marked with TODO.

  4. Install MLAgentBench

    main

    You can install the MLAgentBench package in editable mode using pip. It is recommended to use Python 3.10. To install all necessary dependencies, run the provided install.sh script.

    pip install -e .
# Then install dependencies
bash install.sh

  5. Run MLAgentBench in a Docker sandbox

    main

    Because agents modify and execute files, it is highly recommended to run experiments within a Docker container.

    1. Pull the official image: docker pull qhwang123/researchassistant:latest

    2. Run the container by mounting your current directory to /MLAgentBench inside the container. This allows the agent to work in a controlled environment while persisting files to your host.

    Windows PowerShell:

    docker run -it --user root -v ${PWD}:/MLAgentBench -w /MLAgentBench qhwang123/researchassistant:latest

    Mac or Linux:

    docker run -it --user root -v "$(pwd)":/MLAgentBench -w /MLAgentBench qhwang123/researchassistant:latest

  6. Configure API keys for LLMs

    main

    MLAgentBench supports several LLM providers. You must place the corresponding API key files in the root directory of the repository (or the directory from which you run your scripts):

    • OpenAI: Create openai_api_key.txt with the format organization:APIkey.
    • Claude: Create claude_api_key.txt.
    • CRFM API: Create crfm_api_key.txt.
    • Gemini Pro: Fill in your PROJECT_ID in LLM.py.
    • HuggingFace: Specify the model using the format huggingface/<org name>/<model name>.

    To use an AutoGPT agent, follow the AutoGPT setup guide.

  8. Download FathomNet-FGVC 2023 images

    main

    Use the download_images.py script to download images for the FathomNet 2023 Competition. The script requires a JSON file specifying the data to download (e.g., training or evaluation sets).

    By default, the script creates an images directory in the current location. To save images to a specific location, use the --outpath flag.

    Prerequisites: You must have the dataset downloaded from Kaggle and maintain the following directory structure:

    ../fathomnet-out-of-sample-detection
    |
    |____multilabel_classification
    |  |____train.csv
    |____object_detection
    |  |____eval.json
    |  |____train.json
    |____category_key.csv
    |____demo_download.ipynb
    |____download_images.py
    |____requirements.txt
    |____sample_submission.csv
    # Download training data
python download_images.py object_detection/train.json

# Download evaluation data
python download_images.py object_detection/eval.json

# Example with custom output path
python download_images.py object_detection/train.json --outpath /path/to/your/images

  9. Run a research agent task (Quick Start)

    main

    To run the research agent on a specific task (e.g., cifar10) using OpenAI models, use the MLAgentBench.runner module. It is critical to redirect stdout/stderr to a log file to capture runtime errors like OOM (Out of Memory).

    Example command using GPT-4 and GPT-3.5-Turbo:

    python -u -m MLAgentBench.runner --python $(which python) --task cifar10 --device 0 --log-dir first_test --work-dir workspace --llm-name gpt-4 --edit-script-llm-name gpt-4 --fast-llm-name gpt-3.5-turbo > first_test/log 2>&1

    Key Arguments:

    • --python: The path to the python executable to use.
    • --task: The name of the task (e.g., cifar10).
    • --log-dir: Directory where logs will be stored.
    • --work-dir: The workspace directory for the agent.
    • --llm-name: The primary LLM to use.
    • --edit-script-llm-name: The LLM used for editing scripts.
    • --fast-llm-name: A faster/cheaper LLM for secondary tasks.
    • --agent_type: Set to Agent to run a baseline (trivial policy).
    python -u -m MLAgentBench.runner --python $(which python) --task cifar10 --device 0 --log-dir first_test  --work-dir workspace --llm-name gpt-4 --edit-script-llm-name gpt-4 --fast-llm-name gpt-3.5-turbo >  first_test/log 2>&1

  10. Evaluate agent performance

    main

    To evaluate the results of agent runs, use the MLAgentBench.eval module. This will process all runs within a log folder and output the results as a JSON file.

    python -m MLAgentBench.eval --log-folder <log_folder> --task <task_name> --output-file <output_name>
    python -m MLAgentBench.eval --log-folder <log_folder>  --task <task_name> --output-file <output_name>