GUI-Actor

Use verifier_data_generation.py to curate training samples from the OS-Atlas dataset. Each sample consists of a positive example (correct target marked with a red circle) and negative examples (either another UI element or a random point).

Run the script for each domain (desktop, mobile, web) using the --root_path of your OS-Atlas data, a --new_directory for saving, and a --file_dict_key corresponding to the domain. Use --selected_size to limit the number of samples.

python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key desktop_domain --selected_size 30000
python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key mobile_domain  --selected_size 30000
python verifier_data_generation.py --root_path ${path_to_OS-Atlas-data} --new_directory ${save_path} --file_dict_key web_domain     --selected_size 30000

To train the Grounding Verifier, you must first organize the OS-Atlas dataset into a specific directory structure. The verifier uses images with a red circle marking candidate positions to predict if a position is correct ("True" or "False") based on a language instruction.

Organize your images into the following mapping:

• windows_splited: ${root_path}/desktop_domain/windows_images
• linux_splited: ${root_path}/desktop_domain/linux_images
• macos_splited: ${root_path}/desktop_domain/macos_images
• widget_captioning: ${root_path}/mobile_domain/combined
• uibert_raw: ${root_path}/mobile_domain/UIBert
• ricosca: ${root_path}/mobile_domain/combined
• amex_raw: ${root_path}/mobile_domain/amex_images
• seeclick_web: ${root_path}/web_domain/seeclick_web_imgs
• fineweb_3m: ${root_path}/web_domain/fineweb

The VisionHead_MultiPatch is a specialized module used to compute attention scores between visual encoder features and decoder query features for grounding tasks.

Mechanism:

1. Self-Attention: Applies self-attention to the encoder hidden states to capture visual context.
2. Projection: Uses projection_enc and projection_dec (linear layers with GELU) to map features to a common space.
3. Scaled Dot-Product: Computes attention scores via torch.matmul(proj_dec, proj_enc.transpose(0, 1)) / sqrt(d_model).
4. Loss Calculation:
   • If labels are provided and do_single_patch=False, it uses KL Divergence between the predicted distribution and the target distribution (normalized labels).
   • If do_single_patch=True and labels are provided, it uses Cross Entropy loss.

# VisionHead_MultiPatch initialization
head = VisionHead_MultiPatch(
    d_model=hidden_size, 
    projection_dim=projection_dim, 
    num_attention_heads=8,
    dropout_rate=0.1
)

# Forward pass
# hidden_state_enc: [n_enc, d_model]
# hidden_state_dec: [n_dec, d_model]
# labels: [n_dec, n_enc] (binary mask)
attn_weights, loss = head(hidden_state_enc, hidden_state_dec, labels=labels)

To use the model, you should load Qwen2_5_VLForConditionalGenerationWithPointer from the gui_actor.modeling_qwen25vl module. The project provides two main model sizes:

• 7B Model: microsoft/GUI-Actor-7B-Qwen2.5-VL (Recommended for GPU usage with flash_attention_2).
• 3B Model: microsoft/GUI-Actor-3B-Qwen2.5-VL (Suitable for CPU or lower-resource environments).

When using a GPU, it is recommended to install flash-attn to enable attn_implementation="flash_attention_2" for better performance.

from transformers import AutoProcessor
from gui_actor.modeling_qwen25vl import Qwen2_5_VLForConditionalGenerationWithPointer
import torch

model_name_or_path = "microsoft/GUI-Actor-7B-Qwen2.5-VL"
data_processor = AutoProcessor.from_pretrained(model_name_or_path)

model = Qwen2_5_VLForConditionalGenerationWithPointer.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
    attn_implementation="flash_attention_2"
).eval()