DOCSEARCHbyAlgolia
Submit Docs

Windows Agent Arena (WAA)

repository·main·Indexed 21 days ago

https://github.com/microsoft/windowsagentarena

A scalable platform for testing and benchmarking multi-modal AI agents within a realistic Windows OS environment. WAA supports reproducible research and large-scale deployment via Azure ML, featuring a Windows 11 VM snapshot and support for various accessibility backends and screen element detection methods like Omniparser. It allows users to run benchmarks locally on WSL/Linux or parallelize them using Azure Machine Learning Compute VMs, and provides a 'Bring Your Own Agent' (BYOA) framework for testing custom agents.

Tokens
22.4K
Snippets
68
Records
101
Agent score
75%

What's inside Windows Agent Arena

  2. Use Interactive Mode to run processes individually

    main

    If you are developing agents or extensions, you can launch the Docker container without automatically starting the VM or client processes. This allows you to control the lifecycle of each component manually.

    1. Launch the container in interactive mode:
      cd scripts
./run-local.sh --interactive true
    2. Once inside the container (which starts with bash), run processes manually:
      • To start the VM: ./start_vm.sh
      • To start the client: ./start_client.sh
    cd scripts
./run-local.sh --interactive true

# Inside the container:
./start_vm.sh
./start_client.sh

  3. Install and use Playwright for Python

    main

    Playwright is used for browser automation.

    Installation:

    pip install playwright
playwright install

    Basic Usage Example:

    from playwright.sync_api import sync_playwright

def run(playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://example.com")
    ## other actions...
    browser.close()

with sync_playwright() as playwright:
    run(playwright)
    from playwright.sync_api import sync_playwright

def run(playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://example.com")
    ## other actions...
    browser.close()

with sync_playwright() as playwright:
    run(playwright)

  4. Deploy Windows Arena locally on WSL or Linux

    main

    Local deployment on WSL or Linux involves two main steps: configuring the configuration file and preparing the Windows Arena Docker image.

    1. Configuration file

    Ensure your configuration file is set up according to the project requirements.

    2. Prepare the Windows Arena Docker Image

    Use the provided build script to prepare the Docker image. If you have made changes to Dockerfile-WinArena-Base, you must use the --build-base-image flag to rebuild the base image locally.

    # Build the image including the base image if Dockerfile-WinArena-Base was changed
./build-container-image.sh --build-base-image true

# View all available build options
./build-container-image.sh --help

  7. Set up Azure resources for Windows Agent Arena

    main

    Follow these steps to prepare your Azure environment:

    1. Create a Resource Group: Create a new group (e.g., agents) in your preferred region.
    2. Create Azure Machine Learning Resource: Within the resource group, create an Azure ML workspace (e.g., agents_ml). Ensure you enable the automatic creation of:
      • Storage Account (Note the name for uploading the golden image).
      • Key vault.
      • Application Insights.
      • [Optional] Container Registry (to store custom Docker images privately).
    3. Configure Startup Script: In the Azure ML portal, navigate to the Notebooks tab. In your user-assigned folder, create a bash file named compute-instance-startup.sh and copy the contents from scripts/azure_files/compute-instance-startup.sh into it. This script applies base configurations to new VMs.
    4. Check Quota: Ensure you have enough compute quota. The benchmark uses Standard_D8_v3 VM sizes (8 cores) which require support for nested virtualization.

  8. Use Dev Mode for local development and testing

    main

    When developing or testing initialization scripts in src/win-arena-container/vm/setup or the Python server in src/win-arena-container/vm/setup/server, use the --mode dev flag.

    In dev mode, a shared folder is mounted between the Docker host and the Windows 11 VM at \host.lan\Data. This allows code changes made to the src/win-arena-container/vm/setup folder on your host machine to be immediately reflected inside the Windows 11 VM, accelerating the development loop.

    # Prepare the image in dev mode
cd ./scripts
./run-local.sh --mode dev --prepare-image true

# Run the full setup including the client
./run-local.sh --mode dev --start-client true

  9. Configure LibreOffice Calc for evaluation

    main

    To evaluate LibreOffice Calc files, ensure the following libraries are installed:

    openpyxl
pandas
lxml
xmltodict

    Generate CSV from XLSX

    You can export specific sheets from an XLSX file to CSV using the following command:

    libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx

    Note: The last 1 in the conversion options specifies the sheet number (starting from 1) to export. Refer to CSV Filter Options for details.

    Use compare_table for XLSX evaluation

    Evaluation of .xlsx files relies on the compare_table function. It accepts two filenames and a list of options (rules).

    Rule Types:

    • sheet_data: Compares internal cell values via pandoc.
    • sheet_print: Compares shown cell values via CSV (requires a generated CSV).

    Sheet Selection (sheet_idx0, sheet_idx1, or sheet_idx):

    • Integer i: Extracts the $i$-th sheet from the result (0-indexed).
    • String with Prefix:
      • RI: Extract from Result xlsx, sheet Index.
      • RN: Extract from Result xlsx, sheet Name.
      • EI: Extract from Expected (golden) xlsx, sheet Index.
      • EN: Extract from Expected (golden) xlsx, sheet Name.

    Matching Rules: Rules can use a structure like {"method": "eq", "ref": "abc"}. These are processed by the utils._match_value_to_rule function.

    libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx

  10. Test Windows 11 VM accessibility from Docker

    main

    After preparing the golden image, you can verify that the Python server has booted (allow up to 1 minute for initialization) and is listening for connections by connecting to the running Docker and sending a request to the screenshot endpoint.

    # Connect to the running docker
cd scripts
./run-local.sh --connect true

# Test the server endpoint
curl -v -X GET http://20.20.20.21:5000/screenshot
# Expected: HTTP/1.1 200 OK

  12. Prepare the Windows 11 Golden Image (WAA Snapshot)

    main

    The 'golden image' is a 30GB Windows 11 VM snapshot that includes all necessary programs and a Python server for agent commands. This setup is performed once.

    1. Download and place the ISO

    1. Download the Windows 11 Enterprise Evaluation (90-day trial, English, United States) ISO from the Microsoft Evaluation Center.
    2. Rename the downloaded file to setup.iso.
    3. Copy it to WindowsAgentArena/src/win-arena-container/vm/image.

    2. Run the automated setup

    Execute the following command to begin the ~20 minute provisioning process:

    cd ./scripts
./run-local.sh --prepare-image true

    Important Notes:

    • Do not interfere with the VM during preparation. It will shut down automatically when finished.
    • You can monitor progress at http://localhost:8006.
    • The resulting image is stored in WindowsAgentArena/src/win-arena-container/vm/storage.
    • If you need to include changes from src/win-arena-container in the image during development, use --skip-build false with run-local.sh (the default is true).
    • If you encounter /bin/bash: bad interpreter: No such file or directory on WSL2, run dos2unix on the scripts:
    cd ./scripts
find . -maxdepth 1 -type f -exec dos2unix {}