repository·main·Indexed 41 days ago
https://github.com/os-copilot/os-genesisA framework for automating the construction of GUI agent trajectory data using reverse task synthesis. OS-Genesis provides tools and models, including versions based on InternVL2 and Qwen2-VL, to train agents for interacting with Android and Web interfaces. It includes pipelines for collecting interaction triples via random walks in AndroidWorld and WebArena, a trajectory reward model for scoring, and evaluation tools for the AndroidControl benchmark.
Transform instructions into trajectories in AndroidWorld by following these steps:
params_new.zip into the android_env/android_world directory, replacing existing folders if necessary.aw_instructions.json) into the android_env/androidworld directory.run_gpt_task_runner.py to generate augmented trajectory information and screenshots.Outputs:
aw_instructions.json: Contains augmented trajectory information.screenshots_gpt_241103).Note on Implementation: The trajectory collection uses inference logic adapted from the M3A agent. Specific modifications are documented via comments within gpt_task_runner.py.
# Move instruction file to the environment directory
# Then run the task runner
python run_gpt_task_runner.pyTo evaluate the AndroidControl Benchmark using OS-Genesis, follow these three steps:
# 1. Clone the repository
git clone https://github.com/OS-Copilot/OS-Genesis.git
# 2. Inference
cd OS-Genesis/evaluation/android_control
bash run_ac_inference.sh $dataset $checkpoint
# 3. Evaluation
python ac_eval.pyTo collect <screen_pre, action, screen_after> triples using a random walk in the AndroidWorld environment, follow these steps:
random_walk_aw.py and run_random_walk_aw.py inside the AndroidWorld directory.EMULATOR_NAME (e.g., AndroidWorldAvd).Outputs include screenshots in both the original format and an allytree annotated format.
# 1. Start the emulator
EMULATOR_NAME=AndroidWorldAvd
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554
# 2. Run the random walk collection
python run_random_walk_aw.pyTo collect <screen_pre, action, screen_after> triples in the WebArena environment:
random_walk_web.py in the WebArena directory.python random_walk_web.pyTo evaluate a trained model using AndroidWorld, you must align the model's action space with the environment. This is achieved by integrating the Genesis agent files into the original AndroidWorld agent directory and replacing the inference script.
Follow these steps:
genesis_agent.py and genesis_utils.py to the directory android_env/android_world/android_world/agents (this is the original path for agents in AndroidWorld).infer.py in android_env/android_world/android_world/agents with your Genesis inference script.run.py script with the android_world suite family.This process follows the standard AndroidWorld framework.
python run.py \
--suite_family=android_world \
--agent_name=genesis_agent \
--perform_emulator_setupThe diversity analysis expects data in specific JSON formats. To prepare data for analysis, you must extract two types of information:
instruction key of each entry.reason field from each step within the steps list.reason_action field from each step within the steps list.Note: When processing trajectories, it is common practice to limit the number of samples (e.g., using a MAX_NUM constant) to maintain computational efficiency during embedding generation.
# Extracting instructions
instructions = [d['instruction'] for d in data_list]
# Extracting Mobile trajectories (using 'reason')
mobile_traj = []
for episode in mobile_data:
if 'steps' in episode:
for step in episode['steps']:
mobile_traj.append(step['reason'])
# Extracting Web trajectories (using 'reason_action')
web_traj = []
for episode in web_data:
if 'steps' in episode:
for step in episode['steps']:
web_traj.append(step['reason_action'])During long data collection runs, the Android Virtual Machine may become unresponsive, likely due to memory issues within the VM.
Workaround: Reset the device directly within Android Studio to release occupied memory and restore normal operation.
If you wish to reproduce the reverse task synthesis process without re-collecting data from emulators, you can use the raw <s_pre, a, s_post> triples provided. These include screenshots and corresponding text (including SoM info).
genesis_rm.py. This serves as a reference for how trajectories are scored.OS-Genesis provides pre-trained models for Android (AndroidControl and AndroidWorld) and Web environments. These models are based on either InternVL2 or Qwen2-VL architectures.
InternVL2-4B. Trained on OS-Genesis-ac-training-data.Qwen2-VL-7B-Instruct. Trained on OS-Genesis-ac-training-data.InternVL2-8B. Trained on OS-Genesis-ac-training-data.InternVL2-4B. Trained on OS-Genesis-aw-training-data.Qwen2-VL-7B-Instruct. Trained on OS-Genesis-aw-training-data.InternVL2-8B. Trained on OS-Genesis-aw-training-data.InternVL2-4B. Trained on OS-Genesis-web-training-data.Qwen2-VL-7B-Instruct. Trained on OS-Genesis-web-training-data.InternVL2-8B. Trained on OS-Genesis-web-training-data.To visually inspect how semantic clusters are distributed in the embedding space, you can project the embeddings and color them by cluster labels generated via HDBSCAN.
Steps:
SentenceTransformer.HDBSCAN(min_cluster_size=2) to the embeddings to identify clusters.from sentence_transformers import SentenceTransformer
from hdbscan import HDBSCAN
import matplotlib.pyplot as plt
# 1. Get embeddings
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode(sentences)
# 2. Cluster
clusterer = HDBSCAN(min_cluster_size=2).fit(embeddings)
# 3. Visualize
plt.scatter(embeddings[:, 0], embeddings[:, 1], c=clusterer.labels_, cmap="Spectral", s=10)
plt.show()