DOCSEARCHbyAlgolia
Submit Docs

SeeClick Documentation

repository·main·Indexed 19 days ago

https://github.com/njucckevin/seeclick

SeeClick is a GUI grounding model built on Qwen-VL designed for downstream agent tasks. It provides workflows for data preprocessing, LoRA fine-tuning, and evaluation on benchmarks such as ScreenSpot, Mind2Web, AITW, and MiniWob. The library supports predicting click points (x, y) and bounding boxes [left, top, right, bottom] as ratios relative to image dimensions.

Tokens
3.5K
Snippets
12
Records
17
Agent score
66%

What's inside SeeClick

  1. Understand the bounding box (bbox) format

    main

    In all GUI grounding datasets used by SeeClick, the target element's position is stored in the bbox key. The format is a list of four decimal numbers: [left, top, right, bottom].

    Each value is a ratio in the range [0, 1] relative to the width or height of the image. For example, a left value of 0.5 indicates the position is halfway across the width of the image.

  2. Install SeeClick for inference

    main

    To use SeeClick for inference, install the required dependencies using pip.

    Note: If you intend to fine-tune the model, you must follow the specific setup instructions for agent tasks and install using requirements_agent.txt instead.

    pip install -r requirements.txt

  3. Prepare SFT data for MiniWob

    main

    After downloading MiniWob screenshots and annotations, place the annotations in the data folder. Use miniwob_process.py to generate the JSON file required for SFT of LVLMs.

    Note: Replace miniwob_imgs with the actual directory path of your downloaded screenshots.

    cd agent_tasks
python miniwob_process.py --imgs_dir miniwob_imgs

  4. Evaluate SeeClick on MiniWob

    main

    Evaluate the fine-tuned model in the MiniWob environment. This evaluation uses the Synapse framework, which requires Chrome and a compatible Chromedriver installed.

    Arguments:

    • --model_path: Path to the trained SeeClick checkpoint.
    • --qwen_path: Path to the original Qwen-VL-Chat checkpoint (for tokenizer/config).
    • --imgs_dir: Directory of downloaded MiniWob screenshots.
    • --num_episodes: Number of evaluation episodes per task.
    • --env_name: Specific task name (default all tests all 55 tasks).
    • --headless: Use headless mode (no GUI) for evaluation.
    cd agent_tasks
python miniwob_test.py --model_path xxxx/SeeClick-miniwob --qwen_path xxxx/Qwen-VL-Chat --imgs_dir miniwob_imgs

  5. Fine-tune SeeClick for downstream agent tasks

    main

    You can fine-tune SeeClick using LoRA. The fine-tuning scripts are based on Qwen-VL but utilize LoRA for customized parameters and support multi-GPU training.

    Use the finetune/finetune_lora_ds.sh script with the following key arguments:

    • --data-path: Path to the generated SFT data (formatted according to Qwen-VL requirements).
    • --qwen-ckpt: Path to the original Qwen-VL checkpoint for the tokenizer.
    • --pretrain-ckpt: The base model for fine-tuning (e.g., SeeClick-pretrain or Qwen-VL).
    • --save-path: Directory where training checkpoints will be stored.
    finetune/finetune_lora_ds.sh --save-name SeeClick_test --max-length 704 --micro-batch-size 4 --save-interval 500 \
    --train-epochs 10 --nproc-per-node 2 --data-path xxxx/data_sft.json --learning-rate 3e-5 \
    --gradient-accumulation-steps 8 --qwen-ckpt xxxx/Qwen-VL-Chat --pretrain-ckpt xxxx/SeeClick-pretrain \
    --save-path xxxx/checkpoint_qwen

  6. Pre-train and evaluate on ScreenSpot

    main

    The repository provides a workflow for processing data, pre-training, and evaluating on the ScreenSpot benchmark.

    1. Data Processing

    Run pretrain_process.py to generate a dataset of approximately 1M samples for continual pre-training. The output is saved to ../data/sft_train.json.

    2. GUI Grounding Pre-training

    Use the finetune/finetune_lora_ds.sh script to perform pre-training on the processed dataset.

    3. Evaluation

    Run screenspot_test.py to evaluate the model on the ScreenSpot benchmark.

    # Data Processing
cd pretrain
python pretrain_process.py --mobile_imgs xxxx/combined --web_imgs xxxx/seeclick_web_imgs \
    --widgetcap_json xxxx/widget_captioning.json --ricosca_json xxxx/ricosca.json \
    --screensum_json xxxx/screen_captioning.json --web_json xxxx/seeclick_web.json \
    --coco_imgs xxxx/coco/train2017 --llava_json xxxx/llava_instruct_150k.jsonl

# Pre-training
cd ..
bash finetune/finetune_lora_ds.sh --save-name seeclick_sft --max-length 768 --micro-batch-size 8 \
    --save-interval 4000 --train-epochs 3 --nproc-per-node 8 --data-path ./data/sft_train.json \
    --learning-rate 3e-5 --gradient-accumulation-steps 1 --qwen-ckpt xxxx/Qwen-VL-Chat \
    --pretrain-ckpt xxxx/Qwen-VL-Chat  --save-path xxxx/checkpoint_qwen

# Evaluation
cd pretrain
python screenspot_test.py --qwen_path xxxx/Qwen-VL-Chat --lora_path xxxx/checkpoint_qwen/seeclick_sft/checkpoint-20000 \
    --screenspot_imgs xxxx/screenspot_imgs --screenspot_test xxxx/ScreenSpot --task all

  8. Fine-tune SeeClick using LoRA

    main

    Fine-tune the pre-trained LVLM using LoRA and multi-GPU training via the finetune_lora_ds.sh script. This process is similar to Qwen-VL fine-tuning but uses LoRA for customized parameters.

    Arguments:

    • --save-name: Name for the fine-tuned model.
    • --max-length: Maximum sequence length.
    • --micro-batch-size: Micro batch size for training.
    • --save-interval: Interval for saving checkpoints.
    • --train-epochs: Number of training epochs.
    • --nproc-per-node: Number of processes per node (for multi-GPU).
    • --data-path: Path to the SFT JSON file generated during preprocessing.
    • --learning-rate: Learning rate.
    • --gradient-accumulation-steps: Number of gradient accumulation steps.
    • --qwen-ckpt: Path to the original Qwen-VL checkpoint for loading the tokenizer.
    • --pretrain-ckpt: Path to the base model (e.g., SeeClick-pretrain or Qwen-VL).
    • --save-path: Directory where training checkpoints will be saved.
    bash finetune/finetune_lora_ds.sh --save-name SeeClick_test --max-length 704 --micro-batch-size 4 --save-interval 500 \
    --train-epochs 10 --nproc-per-node 2 --data-path xxxx/mind2web_train_sft.json --learning-rate 3e-5 \
    --gradient-accumulation-steps 8 --qwen-ckpt xxxx/Qwen-VL-Chat --pretrain-ckpt xxxx/SeeClick-pretrain \
    --save-path xxxx/checkpoint_qwen

  9. Acquire Web GUI pre-training data

    main

    Web training data consists of crawled webpage screenshots and element annotations.

    • Web Screenshots: The full dataset (270k images, 130G) is available here. A smaller subset of 10,000 images is available here.
    • Annotations: The element annotations and text are available here.

    Annotation Schema per sample:

    • img_filename: The interface screenshot file.
    • url: The URL of the webpage.
    • elements: A list of target elements, where each element contains:
      • instruction: Automatically crawled text/instruction for the element.
      • bbox: The bounding box of the target element.
      • data_type: Either "text" or "hover".

  10. Acquire Mobile GUI pre-training data

    main

    SeeClick uses several mobile-specific datasets for pre-training. You can download the specific subsets used for SeeClick training via the provided links:

    • RICO Images: The base images for mobile data can be downloaded from the RICO website or via the provided zip file.
    • Widget Captioning: Contains img_filename, instruction, and bbox. Download here.
    • RICOSCA: Automatically labeled using Android VH. Contains img_filename, instruction, and bbox. Download here.
    • Screen Summarization: Contains img_filename and a list of captions. Download here.

  11. Prepare SFT data for AITW

    main

    After downloading AITW screenshots and annotations, place the annotations in the data folder. Use aitw_process.py to generate the JSON file required for SFT of LVLMs.

    Note: Replace aitw_imgs with the actual directory path of your downloaded screenshots.

    cd agent_tasks
python aitw_process.py --imgs_dir aitw_imgs

  12. Prepare SFT data for Mind2Web

    main

    After downloading Mind2Web screenshots and annotations, place the annotations in the data folder. Use mind2web_process.py to generate the JSON file required for SFT (Supervised Fine-Tuning) of LVLMs.

    Note: Replace mind2web_imgs with the actual directory path of your downloaded screenshots.

    cd agent_tasks
python mind2web_process.py --imgs_dir mind2web_imgs