VisualWebArena

repository·main·Indexed 19 days ago

https://github.com/web-arena-x/visualwebarena

A benchmark for evaluating multimodal autonomous language agents on complex, realistic web-based visual tasks. It provides environments, agent trajectories, and evaluation scripts to measure agent navigation and interaction using visual and textual information. The project includes setup guides for both VisualWebArena and standard WebArena environments via AWS AMI or manual Docker installations for sites including Shopping, Forum, Classifieds, and Wikipedia.

Tokens: 4.7K
Snippets: 15
Records: 15
Agent score: 17%

What's inside VisualWebArena

Set up Social Forum Website (Reddit) manually

main

Download the image tar from Google Drive, Archive.org, or Metis.
Load and run:

docker load --input postmill-populated-exposed-withimg.tar
docker run --name forum -p 9999:80 -d postmill-populated-exposed-withimg

Access via http://<your-server-hostname>:9999/.

docker load --input postmill-populated-exposed-withimg.tar
docker run --name forum -p 9999:80 -d postmill-populated-exposed-withimg

Reset VisualWebArena (VWA) environment

main

To return the VWA environment to its initial state after evaluation, run:

### For VWA:
bash scripts/reset_reddit.sh
bash reset_shopping.sh
curl -X POST http://<your-server-hostname>:9980/index.php?page=reset -d "token=4b61655535e7ed388f0d40a93600254c"

bash scripts/reset_reddit.sh
bash reset_shopping.sh
curl -X POST http://<your-server-hostname>:9980/index.php?page=reset -d "token=4b61655535e7ed388f0d40a93600254c"

Install VisualWebArena
main
To install VisualWebArena, use Python 3.10 or 3.11 (note: Python 3.12 is not supported due to distutils deprecation). Follow these steps to set up a virtual environment and install dependencies:
1. Create and activate a virtual environment.
2. Install requirements via pip.
3. Install Playwright browsers.
4. Install the package in editable mode.
You can verify the installation by running the unit tests with pytest -x.
```
# Python 3.10 (or 3.11, but not 3.12 cause 3.12 deprecated distutils needed here)
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
playwright install
pip install -e .

# Verify installation
pytest -x
```
https://github.com/web-arena-x/visualwebarena/blob/main/README.md

Disable Magento product re-indexing for Shopping Website

main

To prevent the Magento shopping site from automatically re-indexing products (which can interfere with evaluation), run the following commands inside the shopping (VWA) or shopping_admin (WebArena) container:

docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalogrule_product
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalogrule_rule
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalogsearch_fulltext
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalog_category_product
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule customer_grid
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule design_config_grid
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule inventory
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalog_product_category
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalog_product_attribute
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalog_product_price
docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule cataloginventory_stock

Note: Replace shopping with shopping_admin if using the WebArena setup.

docker exec shopping /var/www/magento2/bin/magento indexer:set-mode schedule catalogrule_product
# ... (repeat for other indexers)

Set up Shopping Website (OneStopShop) manually

main

Download the image tar from Google Drive, Archive.org, or Metis.
Load and run:

docker load --input shopping_final_0712.tar
docker run --name shopping -p 7770:80 -d shopping_final_0712
# Wait ~1 min
docker exec shopping /var/www/magento2/bin/magento setup:store-config:set --base-url="http://<your-server-hostname>:7770"
docker exec shopping mysql -u magentouser -pMyPassword magentodb -e 'UPDATE core_config_data SET value="http://<your-server-hostname>:7770/" WHERE path = "web/secure/base_url";'
docker exec shopping /var/www/magento2/bin/magento cache:flush
# (Optional) Disable re-indexing as described in the re-indexing guide.

Access via http://<your-server-hostname>:7770.

docker load --input shopping_final_0712.tar
docker run --name shopping -p 7770:80 -d shopping_final_0712
docker exec shopping /var/www/magento2/bin/magento setup:store-config:set --base-url="http://<your-server-hostname>:7770"
docker exec shopping mysql -u magentouser -pMyPassword magentodb -e 'UPDATE core_config_data SET value="http://<your-server-hostname>:7770/" WHERE path = "web/secure/base_url";'
docker exec shopping /var/www/magento2/bin/magento cache:flush

Set up WebArena using AWS AMI

main

To set up the standard WebArena environment using the pre-installed AMI:

Launch EC2 Instance: Use AMI ID ami-080f6d73cfce497a1 in region us-east-2. Recommended instance type is t3a.xlarge with a 1000GB EBS root volume.
Networking: Create a security group allowing all inbound traffic and associate an Elastic IP to get a static hostname (<your-server-hostname>).
Start Docker Services:

docker start gitlab
docker start shopping_admin
cd /home/ubuntu/openstreetmap-website/
docker compose start

Initialize Data: Wait ~1 minute, then run:

docker exec shopping_admin /var/www/magento2/bin/magento setup:store-config:set --base-url="http://<your-server-hostname>:7780"
docker exec shopping_admin mysql -u magentouser -pMyPassword magentodb -e 'UPDATE core_config_data SET value="http://<your-server-hostname>:7780/" WHERE path = "web/secure/base_url";'
docker exec shopping_admin /var/www/magento2/bin/magento cache:flush
docker exec gitlab sed -i "s|^external_url.*|external_url 'http://<your-server-hostname>:8023'|" /etc/gitlab/gitlab.rb
docker exec gitlab gitlab-ctl reconfigure

### For WebArena:
docker start gitlab
docker start shopping_admin
cd /home/ubuntu/openstreetmap-website/
docker compose start

Reset WebArena environment

main

To return the WebArena environment to its initial state, you must remove the existing containers and re-run them:

### For WebArena:
docker stop shopping_admin gitlab
docker remove shopping_admin gitlab
docker run --name shopping_admin -p 7780:80 -d shopping_admin_final_0719
docker run --name gitlab -d -p 8023:8023 gitlab-populated-final-port8023 /opt/gitlab/embedded/bin/runsvdir-start
# After running these, repeat the configuration steps (setting base-url and reconfiguring gitlab) from the initial setup guide.

docker stop shopping_admin gitlab
docker remove shopping_admin gitlab
docker run --name shopping_admin -p 7780:80 -d shopping_admin_final_0719
docker run --name gitlab -d -p 8023:8023 gitlab-populated-final-port8023 /opt/gitlab/embedded/bin/runsvdir-start

Run End-to-end Evaluation
main
To perform a full evaluation, follow these steps in order:
1. Setup Environments: Ensure standalone environments are running (see environment_docker/README.md).
2. Configure URLs: Set the environment variables as described in the configuration guide.
3. Generate Config Files: Run python scripts/generate_test_data.py to create *.json files in the ./config_files directory.
4. Prepare Cookies: Run bash prepare.sh to obtain and save auto-login cookies for all websites.
5. Set API Keys:
  For OpenAI: export OPENAI_API_KEY=your_key
  For Gemini: Authenticate via gcloud auth login and gcloud config set project <your_project_name>.
6. Launch Evaluation: Use python run.py with the appropriate flags for your model and instruction path.
```
# Example: Reproduce GPT-3.5 captioning baseline
python run.py \
  --instruction_path agent/prompts/jsons/p_cot_id_actree_3s.json \
  --test_start_idx 0 \
  --test_end_idx 1 \
  --result_dir <your_result_dir> \
  --test_config_base_dir=config_files/vwa/test_classifieds \
  --model gpt-3.5-turbo-1106 \
  --observation_type accessibility_tree_with_captioner
```
https://github.com/web-arena-x/visualwebarena/blob/main/README.md

Set up VisualWebArena (VWA) using AWS AMI

main

The recommended way to start the VisualWebArena environment is using a pre-installed Amazon Machine Image (AMI).

Launch EC2 Instance: Use AMI ID ami-080f6d73cfce497a1 in region us-east-2. Recommended instance type is t3a.xlarge with a 1000GB EBS root volume.
Networking: Create a security group allowing all inbound traffic and associate an Elastic IP to get a static hostname (<your-server-hostname>).
Start Docker Services: Log into the instance and run:

docker start shopping
docker start forum
docker start kiwix33
cd classifieds_docker_compose
# Edit classifieds_docker_compose/docker-compose.yml to set CLASSIFIEDS to http://<your-server-hostname>:9980/
docker compose up --build -d

Initialize Data: Wait ~1 minute for services to start, then run:

docker exec classifieds_db mysql -u root -ppassword osclass -e 'source docker-entrypoint-initdb.d/osclass_craigslist.sql'
docker exec shopping /var/www/magento2/bin/magento setup:store-config:set --base-url="http://<your-server-hostname>:7770"
docker exec shopping mysql -u magentouser -pMyPassword magentodb -e 'UPDATE core_config_data SET value="http://<your-server-hostname>:7770/" WHERE path = "web/secure/base_url";'
docker exec shopping /var/www/magento2/bin/magento cache:flush

### For VWA:
docker start shopping
docker start forum
docker start kiwix33
cd classifieds_docker_compose
vi classifieds_docker_compose/docker-compose.yml  # Set CLASSIFIEDS to your site url `http://<your-server-hostname>:9980/`, and change the reset token if required
docker compose up --build -d

Set up Wikipedia Website manually
main
1. Download the Wikipedia data from Google Drive, Archive.org, or Metis.
2. Run using kiwix-serve:
```
docker run -d --name=wikipedia --volume=<your-path-to-downloaded-folder>/:/data -p 8888:80 ghcr.io/kiwix/kiwix-serve:3.3.0 wikipedia_en_all_maxi_2022-05.zim
```
Access via http://<your-server-hostname>:8888/wikipedia_en_all_maxi_2022-05/A/User:The_other_Kiwix_guy/Landing.
```
docker run -d --name=wikipedia --volume=<your-path-to-downloaded-folder>/:/data -p 8888:80 ghcr.io/kiwix/kiwix-serve:3.3.0 wikipedia_en_all_maxi_2022-05.zim
```
https://github.com/web-arena-x/visualwebarena/blob/main/environment_docker/README.md

Host the WebArena Homepage

main

The homepage provides a central navigation point for all available websites. To host it:

Configure Hostname: Replace the <your-server-hostname> placeholder in webarena-homepage/templates/index.html with your actual server hostname using perl:

YOUR_ACTUAL_HOSTNAME="your.hostname.com"
YOUR_ACTUAL_HOSTNAME=${YOUR_ACTUAL_HOSTNAME%/}
perl -pi -e "s|<your-server-hostname>|${YOUR_ACTUAL_HOSTNAME}|g" webarena-homepage/templates/index.html

Run Flask Server:

cd webarena_homepage
flask run --host=0.0.0.0 --port=4399

Access via http://<your-server-hostname>:4399.

YOUR_ACTUAL_HOSTNAME="your.hostname.com"
YOUR_ACTUAL_HOSTNAME=${YOUR_ACTUAL_HOSTNAME%/}
perl -pi -e "s|<your-server-hostname>|${YOUR_ACTUAL_HOSTNAME}|g" webarena-homepage/templates/index.html

cd webarena_homepage
flask run --host=0.0.0.0 --port=4399

Set up Classifieds Website manually

main

Download the image zip from Google Drive or Archive.org.
Setup and run:

unzip classifieds_docker_compose.zip
cd classifieds_docker_compose
# Edit classifieds_docker_compose/docker-compose.yml to set CLASSIFIEDS to http://<your-server-hostname>:9980/
docker compose up --build -d
# Wait ~1 min
docker exec classifieds_db mysql -u root -ppassword osclass -e 'source docker-entrypoint-initdb.d/osclass_craigslist.sql'

Access via http://<your-server-hostname>:9980.

unzip classifieds_docker_compose.zip
cd classifieds_docker_compose
docker compose up --build -d
docker exec classifieds_db mysql -u root -ppassword osclass -e 'source docker-entrypoint-initdb.d/osclass_craigslist.sql'