WebVoyager Documentation

repository·main·Indexed 22 days ago

https://github.com/minorjerry/webvoyager

An end-to-end web agent powered by Large Multimodal Models (LMMs) that navigates and interacts with real-world websites using Selenium. It includes tools for automated evaluation using GPT-4V, a system for expanding task pools via GPT-4, and support for both visual and text-only (accessibility tree) observation modes.

Tokens
2.5K
Snippets
5
Records
6
Agent score
28%

What's inside WebVoyager

  1. How to develop and optimize prompts

    main

    The agent's performance is heavily dependent on the system prompt. The default prompt is located in prompts.py.

    Optimization Strategies:

    • Generality: Try to optimize the prompt without adding site-specific instructions to maintain the agent's generalist nature.
    • Site-Specific Tuning: If you are only targeting specific websites, you may add specific instructions to the prompt.
    • Action Customization: To change how the agent performs actions or to add new action types, you can design a custom extract_information function to parse the model's output and modify run.py to implement the new execution logic.
  2. Setup the WebVoyager environment

    main

    To run WebVoyager, you need to set up a Python 3.10 environment and ensure Chrome is installed. If running on a Linux server (e.g., CentOS), install chromium-browser via yum.

    Follow these steps to create a conda environment and install dependencies:

    1. Create and activate a new conda environment.
    2. Install the required packages using pip.

    Note: The latest version of Selenium does not require a separate ChromeDriver installation if Chrome is present.

    conda create -n webvoyager python=3.10
    conda activate webvoyager
    pip install -r requirements.txt
  3. Run WebVoyager agent

    main

    To run the agent, prepare your test tasks in data/tasks_test.jsonl. For time-sensitive tasks (like Booking or Google Flights), manually update the dates in the task description to ensure they are current.

    1. Populate data/tasks_test.jsonl with your desired tasks.
    2. Update the api_key in your execution script.
    3. Execute the run script.

    By default, the agent uses visual information. You can also run a 'text-only' mode where observations are limited to the accessibility tree.

    # Standard multimodal run
    # Edit run.sh first to include your API key
    bash run.sh
    
    # Example of the underlying command used in run.sh
    nohup python -u run.py \
        --test_file ./data/tasks_test.jsonl \
        --api_key YOUR_OPENAI_API_KEY \
        --headless \
        --max_iter 15 \
        --max_attached_imgs 3 \
        --temperature 1 \
        --fix_box_color \
        --seed 42 > test_tasks.log &
  4. Perform GPT-4V based auto-evaluation

    main

    WebVoyager provides an automated evaluation protocol using GPT-4V to judge if an agent successfully completed a task based on the task description and the last $k$ screenshots.

    To run the evaluation:

    1. Navigate to the evaluation directory.
    2. Update the api_key and process_dir in evaluation/run_eval.sh.
    3. Execute bash run_eval.sh.

    Note: The process_dir should point to the directory containing your experiment results (e.g., ../results/examples).

    # Example command inside evaluation/run_eval.sh
    nohup python -u auto_eval.py \
        --api_key YOUR_OPENAI_API_KEY \
        --process_dir ../results/examples \
        --max_attached_imgs 15 > evaluation.log &
  5. Expand tasks using GPT-4

    main

    You can expand your task pool by using GPT-4 to generate new tasks based on existing examples. Use a prompt that provides in-context examples of <TASK: xxx; WEB: xxx;> and instructs the model to generate new tasks for a specific website.

    Prompting Guidelines:

    1. Do not include video viewing requirements.
    2. For future dates (bookings/flights), specify a range (e.g., <date 1> to <date 2>).
    3. Ensure tasks have clear goals and avoid overly complex page operations.
    4. For real-time info (news), avoid requesting massive historical data to prevent excessive scrolling.
    5. Use the format: TASK: {Generated-task}|||WEB: {Website-name, https-address}
    Here are some example tasks and the websites that need to be interacted with to solve these tasks.
    """
    <TASK: xxx; WEB: xxx;>
    <other in-context examples>
    """
    
    Please carefully analyze the above TASKs and then generate new TASKs for {website}. Please use diverse descriptions and do not repeat the task descriptions in examples.
    
    Pay attention:
    1. Do not include the requirement to view videos in the task.
    2. In the generated task, if you need to declare a specific date in the future (such as booking, flights ...), you can choose the date in the range of <date 1> to <date 2>.
    3. The generated task should have a clear goal and should not require complicated web page operations.
    4. When looking for real-time information in the past (such as ArXiv, BBC News ...), don't ask for too much information for a certain period of time in the past, as this requires a lot of web scrolling and page flipping. But you may request information for certain points in time, e.g. latest.
    5. To improve randomness and diversity, please try not to repeat entities that were asked about in examples.
    
    Think carefully about the functions of given websites, and please note that the generated TASK can be solved by the corresponding website. The format of the answer must be: TASK: {Generated-task}|||WEB: {Website-name, https-address}
  6. Reference: WebVoyager CLI arguments

    main

    The run.py script accepts several parameters to control the agent's behavior, model selection, and web navigation settings.

    ### General
    - `--test_file`: The task file to be evaluated (format found in `data/`).
    - `--max_iter`: Maximum number of online interactions per task. Failure occurs if this limit is reached.
    - `--api_key`: Your OpenAI API key.
    - `--output_dir`: Directory to save the browsing trajectory.
    - `--download_dir`: Directory for files downloaded by the agent (e.g., PDFs).
    
    ### Model
    - `--api_model`: The model used for decisions (e.g., `gpt-4-vision-preview`). For text-only, use models like `gpt-4-1106-preview`.
    - `--seed`: Seed for reproducibility (Beta).
    - `--temperature`: Controls model diversity.
    - `--max_attached_imgs`: Number of recent screenshots to keep in context (context clipping).
    - `--text_only`: If set, observations will be the accessibility tree instead of images.
    
    ### Web Navigation
    - `--headless`: Runs without an explicit browser window (recommended for Linux servers). Note: This affects screenshot size as the address bar is absent.
    - `--save_accessibility_tree`: Whether to save the Accessibility Tree for the current page.
    - `--force_device_scale`: Sets device scale factor to 1 (recommended when using accessibility trees).
    - `--window_width`: Width of the browser window (default: 1024).
    - `--window_height`: Height of the browser window (default: 768).
    - `--fix_box_color`: Uses GPT-4-ACT to overlay bounding boxes on interactive elements. If set, box color is fixed to black; otherwise, it is random.