DOCSEARCHbyAlgolia
Submit Docs

Mobile-Agent Documentation

repository·main·Indexed 27 days ago

https://github.com/x-plug/mobileagent

A family of multi-modal GUI agents developed by Tongyi Lab (Alibaba Group) for automating tasks across mobile, desktop, and web interfaces. The project includes Mobile-Agent-v1, Mobile-Agent-v2, and Mobile-Agent-E, supporting Android and Harmony OS via ADB. It features integration with models like GUI-Owl and Qwen-VL, a screenshot-based inference API, and the Mobile-Eval benchmark for evaluating agent performance across various single-app and multi-app scenarios.

Tokens
28.2K
Snippets
81
Records
149
Agent score
94%

What's inside Mobile-Agent

  1. Overview of WebJudge automatic evaluator

    main

    WebJudge is an LLM-as-a-Judge evaluation method designed for online environments. It consists of three components:

    1. Key Point Identification: Identifies necessary points for task completion based on instructions.
    2. Key Screenshot Identification: Selects important screenshots from the agent's trajectory to provide visual evidence while reducing token usage.
    3. Outcome Judgment: Produces a judgment based on the task description, identified key points, key screenshots, and the action history.

  2. Overview of Mobile-Agent Project Series

    main

    The Mobile-Agent repository contains a series of research works and agent frameworks:

    • Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents.
    • Mobile-Agent-v3: Multi-modal and multi-platform GUI agent framework.
    • UI-S1: GUI Automation via Semi-online Reinforcement Learning.
    • GUI-Critic-R1: A GUI-Critic for pre-operative error diagnosis.
    • PC-Agent: Multi-agent framework for multimodal PC operation.
    • Mobile-Agent-E: Multi-agent for self-evolving mobile phone operation.
    • Mobile-Agent-v2: Multi-agent for multimodal mobile phone operation.
    • Mobile-Agent-v1: Single-agent for multimodal mobile phone operation.

  3. GUI-Owl 1.5 Model Family Overview

    main

    GUI-Owl 1.5 is a next-generation native GUI agent model family built on Qwen3-VL. It supports multi-platform GUI automation across desktops, mobile devices, and browsers.

    Available Models:

    • GUI-Owl-1.5-2B-Instruct: Fast inference/edge deployment.
    • GUI-Owl-1.5-4B-Instruct
    • GUI-Owl-1.5-8B-Instruct
    • GUI-Owl-1.5-8B-Thinking: For complex tasks requiring planning and reflection.
    • GUI-Owl-1.5-32B-Instruct
    • GUI-Owl-1.5-32B-Thinking

    Key Capabilities:

    • Tool & MCP calling: Native support for external tool invocation and MCP server coordination.
    • Long-horizon memory: Built-in memory capability without external orchestration.
    • Multi-agent ready: Can act as a standalone agent or specialized roles (planner, executor, verifier, notetaker).

  4. Explore the Mobile-Agent Series of Work

    main

    The Mobile-Agent repository contains several specialized agent frameworks and models for GUI automation across different platforms:

    • Mobile-Agent-v3.5: A unified multi-platform GUI agent (Preprint).
    • Mobile-Agent-v3: A multimodal, multi-platform GUI agent (Preprint).
    • UI-S1: Advances GUI automation through semi-online reinforcement learning.
    • GUI-Critic-R1: A GUI-Critic method for a priori error diagnosis (NeurIPS 2025).
    • PC-Agent: A multi-agent system for multimodal PC operations (ICLR 2025 Workshop).
    • Mobile-Agent-E: A multi-agent system for self-evolving mobile operations (Preprint).
    • Mobile-Agent-v2: A multi-modal mobile operation agent (NeurIPS 2024).
    • Mobile-Agent-v1: A single agent for multimodal mobile operations (ICLR 2024 Workshop).

  5. Understand the RLHF dataset format

    main

    The RLHF (Reinforcement Learning from Human Feedback) datasets are stored in Parquet files. Data is organized using a chat-based prompt format to support multi-turn conversations. Prompts may include specific instruction-following text to guide the model toward a particular output format, which facilitates automated answer extraction.

    Each data entry typically includes:

    • data_source: The origin of the data (e.g., openai/gsm8k).
    • prompt: A list of chat messages with role and content keys.
    • ability: The capability being tested (e.g., math).
    • reward_model: Configuration for evaluating the response, including the style (e.g., rule) and the ground_truth values.

  6. Add a new Huggingface model to verl

    main

    To integrate a new Huggingface model into verl, follow these steps to ensure it is optimized for parallelization and uses packed inputs:

    1. Copy Model Files: Create a new file under verl/models/hf and copy only the specific model file from the huggingface/transformers/models directory into that location.
    2. Optimize for Packed Inputs: Modify the model file to remove inference-related code (such as KV cache). Update the inputs to include only:
      • input_ids (shape: total_nnz,)
      • cu_seqlens (shape: total_nnz + 1,)
      • max_seqlen_in_batch: int Note: This modification requires using Flash Attention with a causal mask.
    3. Verify with Tests: Add a test case in tests/models/hf to compare the output of your modified verl version against the original Huggingface version.
    4. Implement Parallelism: Implement functions to support Tensor Parallelism, Data Parallelism, and Pipeline Parallelism (see details below).

  7. Prepare mobile device connection via ADB

    main

    Mobile-Agent-v2 supports Android and HarmonyOS (version <= 4). Follow these steps to connect your device:

    1. Download Android Debug Bridge (ADB).
    2. Enable "USB debugging" or "ADB debugging" in your device's Developer Options. For HyperOS, you must also enable "USB debugging (Security settings)".
    3. Connect the device to your computer via USB and select "File Transfer" mode.
    4. Verify the connection by running:
      /path/to/adb devices
      If the device list is not empty, the connection is successful.
    5. MacOS/Linux users: Grant execution permissions to ADB:
      sudo chmod +x /path/to/adb

    Note on paths:

    • Windows: Use the format xx/xx/adb.exe.
    • MacOS/Linux: Use the format xx/xx/adb.
    /path/to/adb devices

  10. Access GUI-Owl-1.5 Models via API

    main

    The GUI-Owl-1.5 series models (including 2B, 4B, 8B, 32B, and 235B variants) are available for online inference through:

    • Alibaba Cloud Bailian: For production-grade inference.
    • Modelscope API-Inference: For testing and development.

    These models support desktop, mobile, and browser automation, featuring capabilities for grounding, tool/MCP calling, and long-horizon memory.

  11. Run Mobile-Agent using a free API

    main

    If you do not have an OpenAI API Key, you can use the provided free API endpoint. Use the run_api.py script with the required URL, token, and instruction.

    Note: To avoid hallucinations in visual perception, it is strongly recommended to use Mobile-Agent with English-only systems and apps.

    python run_api.py --adb_path /path/to/adb --url "The url you got" --token "The token you got" --instruction "your instruction"