Mind2Web Dataset and Framework

The dataset is organized into training and various test splits (Cross Task, Cross Website, and Cross Domain). After cloning the training data and unzipping the test data, the directory structure should follow this pattern:

Mind2Web
├── data
│   ├── train
│   │   └── train_*.json
│   ├── test_task
│   │   └── test_task_*.json
│   ├── test_website
│   │   └── test_website_*.json
│   └── test_domain
│       └── test_domain_*.json
└── ...

To evaluate using Large Language Models (LLMs), you need the dataset, candidate generation results, and an OPENAI_API_KEY. The evaluation uses a multi-choice QA formulation where the model selects the target element from a list of candidates.

Setup

1. Set your OpenAI API key in the environment: export OPENAI_API_KEY={YOUR_API_KEY}
2. Use the 3-shot prompt located at src/action_prediction/llm_prompt.json.

Execution

Run the evaluation script using the following command:

export OPENAI_API_KEY={YOUR_API_KEY}
python action_prediction/evaluate_llm.py \
  +output_path={OUTPUT_DIR} \
  +llm_prompt=action_prediction/llm_prompt.json \
  +llm=gpt-3.5-turbo \
  +llm_rate_limit=60 \
  +top_k=50

To evaluate trained action prediction models (such as flan-t5-base, flan-t5-large, or flan-t5-xl), you must first configure the parameters in action_prediction/conf/config.yaml. The project uses Hydra for configuration management.

Configuration Keys

In action_prediction/conf/config.yaml, ensure the following are set:

• data.data_path: The directory where you cloned the Mind2Web dataset.
• data.score_file: The path to the downloaded pickle file containing candidate generation model outputs.
• hydra.run.dir: The working directory for log files.

Execution

Run the evaluation script using the following command structure:

python action_prediction/evaluate.py \
  +model_path={MODEL_PATH_OR_NAME} \
  model=flan-t5-large \
  +output_path={OUTPUT_PATH} \
  +top_k=50

Mind2Web provides a training set available directly on Huggingface and a protected test set to prevent data contamination.

1. Training Set: Clone the repository from Huggingface using git.
2. Test Set: Download the test split zip files from Huggingface. Unzip them into the same directory as your training data using the password mind2web.

Important: Do not redistribute unzipped data files online. The dataset contains canary GUIDs to detect unauthorized use in training corpora.

git clone git@hf.co:datasets/osunlp/Mind2Web

Fine-tuning is performed using a seq2seq T5 model implementation based on Huggingface Transformers's Seq2SeqTrainer. Configuration is managed via action_prediction/conf/config.yaml. Checkpoints are saved in the workdir defined by Hydra.

Use torchrun for distributed training. Example for 4 GPUs:

torchrun --nproc-per-node 4 --master_port=8004\
    action_prediction/train.py\
    model=flan-t5-large\
    train.per_device_train_batch_size=8\
    train.gradient_accumulation_steps=1\
    train.fsdp=True\
    train.num_gpus=4\
    train.epoch=5\
    run_id="full"

The raw dump contains high-fidelity data including Playwright trace files, network traffic (.har), video recordings, and various page snapshots (.mhtml).

Due to its size, the raw dump is shared via Globus. You can access the collection via the following link: https://app.globus.org/file-manager?origin_id=32e6b738-a0b0-47f8-b475-26bf1c5ebf19

Key files in the raw dump structure:

• trace.zip: Playwright trace file (inspect via trace viewer).
• session.har.zip: Network traffic for replaying.
• videos/: WebM recordings of the annotation process.
• dom_content.json: DOM snapshots extracted via DOMSnapshot.captureSnapshot.
• screenshot.json: Page screenshots stored as base64 strings.
• {action_id}_before/after.mhtml: Standalone MHTML snapshots of the page state.

Because different annotators might select slightly different nodes for the same logical action, the project provides heuristics to normalize the selection:

1. move_up_annotation: Finds the closest clickable ancestor that contains the target interaction position.
2. move_down_annotation: Finds all descendants within the target bounding box.
3. set_alternative: Sets data_pw_testid_buckeye_alt on the normalized ancestor and data_pw_testid_buckeye_alt_fuzzy on the candidate descendants to allow for flexible matching.

The project uses a build_dom_tree function to reconstruct a full HTML tree from a DOMSnapshot.NodeTreeSnapshot object. This is necessary because the raw data is stored in a compressed/indexed format (nodes, layout, and strings) rather than raw HTML.

Key steps in reconstruction:

1. Map indices to actual string values using str_mapping.
2. Iterate through nodes to create lxml.etree.Element objects.
3. Apply attributes like bounding_box_rect, text_value, input_value, and is_clickable.
4. Recursively handle iframes via contentDocumentIndex.
5. Reconstruct the parent-child hierarchy using parentIndex.

The processing pipeline includes logic to prune the DOM tree of nodes that do not contribute to user interaction or visibility.

• is_visible: Checks if a node's bounding_box_rect is valid (non-negative coordinates and positive dimensions).
• is_empty: Determines if a node has no text, no meaningful attributes (from salient_attributes), and no children.
• Special Handling: Elements like a, button, select, option, and input are treated as visible if they contain text, even if their bounding box suggests otherwise.
• clean_empty_node: Iterates through the tree in reverse to safely remove nodes identified as empty or invisible.