WebCanvas Framework

The WebCanvas base agent framework is designed to be modular and plug-and-play. Developers can iterate on LLM-based web agents by implementing or swapping the following key modules:

• Planning: Logic for determining the next steps in a task.
• Observation: Capturing the current state of the web environment.
• Memory: Storing historical context and observations.
• Reward: Providing feedback based on agent actions.
• Action Execution: Performing actions within the web environment.
• Evaluation: Assessing the agent's performance against task goals.

The Mind2web-live leaderboard evaluates web agents based on two primary performance metrics:

• Completion rate: The success rate of the model in completing the assigned tasks.
• Task success rate: The success rate of the model in performing the specific steps or actions required by the tasks.

Agents marked with a star (★) indicate that the model or the underlying code has been open-sourced and verified.

WebCanvas automatically calculates token consumption for the planning and global reward modules using tiktoken. Results are saved in the token_results folder as JSON.

Custom Pricing: To calculate monetary expenditure for models not listed in the default token_pricing in configs/setting.toml:

1. Add the full model name (e.g., gpt-4o-2024-05-13) to the pricing_models list.
2. Define the unit prices using the format: {model_name}_input_price = <price> and {model_name}_output_price = <price>.

To use Gemini models within WebCanvas, set the GOOGLE_API_KEY environment variable. Replace your-api-key-here with your actual key from Google.

# MacOS/Linux
export GOOGLE_API_KEY='your-api-key-here'

# Windows
setx GOOGLE_API_KEY "your-api-key-here"

Execute the evaluation script using evaluate.py. You can customize the reward mode, the data index, the specific task, and the models used for planning and reward reasoning.

Configuration: The task_mode (batch vs. single) and the path to your test file are controlled via configs/setting.toml.

python evaluate.py \
    --global_reward_mode dom_reward \
    --index -1 \
    --single_task_name "Find Dota 2 game and add all DLC to cart in steam." \
    --planning_text_model gpt-4o-mini \
    --global_reward_text_model gpt-4o-mini

To prepare your environment for evaluating web agents, follow these steps to install Python and Node.js dependencies:

1. Python Environment: Create and activate a Conda environment with Python 3.11 and install the required packages.
2. Node.js Dependencies: Initialize npm and install axios.
3. API Keys: Ensure you have set up required LLM API keys (refer to agent/LLM/README.md) and Google Search credentials to prevent GUI agent blocking.

Recommended Environment for Mind2Web-Live: For optimal performance, use a Windows server with Chrome or Firefox, preferably located in the United States.

# Python setup
conda create -n webcanvas python=3.11
conda activate webcanvas
pip install -r requirements.txt

# Node.js setup
npm init -y
npm install axios

# Google Search API setup
export GOOGLE_API_KEY=your_api_key
export GOOGLE_CX=your_custom_search_engine_id

To participate in a challenge, you must upload your generated out.json file using the dataset_io.py utility.

python data/dataset_io.py upload \
    --file-path /path/to/your/file \
    --challenge-id your_challenge_id \
    --name your_agent_name \
    --base-model your_agent_base_model

You can leverage Browserbase to run headless browsers at scale. To configure it:

1. Get your API Key from the Browserbase Dashboard Settings.
2. Add the key to your .env file as BROWSERBASE_API_KEY.

Alternatively, you can set it as an environment variable for a single command: BROWSERBASE_API_KEY=<your_api_key> <command>

export BROWSERBASE_API_KEY=your_api_key

To use OpenAI models within WebCanvas, set the OPENAI_API_KEY environment variable. Replace your-api-key-here with your actual key from the OpenAI platform.

# MacOS/Linux
export OPENAI_API_KEY='your-api-key-here'

# Windows
setx OPENAI_API_KEY "your-api-key-here"

To use Together AI models within WebCanvas, set the TOGETHER_API_KEY environment variable. Replace your-api-key-here with your actual key from Together AI.

# MacOS/Linux
export TOGETHER_API_KEY='your-api-key-here'

# Windows
setx TOGETHER_API_KEY "your-api-key-here"

To use Claude models within WebCanvas, set the ANTHROPIC_API_KEY environment variable. Replace your-api-key-here with your actual key from Anthropic.

# MacOS/Linux
export ANTHROPIC_API_KEY='your-api-key-here'

# Windows
setx ANTHROPIC_API_KEY "your-api-key-here"