Open-AutoGLM Mobile Intelligent Assistant Framework

Phone Agent is a mobile intelligent assistant framework built on AutoGLM. It uses vision-language models to understand phone screen content and controls devices via ADB (Android Debug Bridge) or HDC (for HarmonyOS).

Key features include:

• Multimodal Perception: Understands screens via vision-language models.
• Intelligent Planning: Generates and executes operation workflows from natural language commands (e.g., "Open eBay and search for wireless earphones").
• Safety: Includes sensitive operation confirmation and supports manual takeover for logins or verification codes.
• Remote Control: Supports remote ADB debugging via WiFi or network.

Open-AutoGLM (Phone Agent) is a mobile intelligent assistant framework built on AutoGLM. It uses a multimodal approach to understand mobile screen content and automates tasks via natural language instructions (e.g., "Open Xiaohongshu and search for food").

Core Capabilities:

• Screen Perception: Uses vision-language models to understand the current UI.
• Task Planning: Automatically parses intent and plans step-by-step actions.
• Device Control: Uses ADB (Android Debug Bridge) or HDC (for HarmonyOS) to execute operations.
• Safety & Control: Includes sensitive operation confirmation and supports manual takeover for login or CAPTCHA scenarios.
• Remote Debugging: Supports remote ADB debugging via WiFi or network connections.

Open-AutoGLM is a phone agent framework that translates natural language instructions into physical device actions.

Workflow:

1. Input: User provides a natural language instruction (e.g., "Open WhatsApp and send a message to John").
2. Perception: The system takes a screenshot of the device.
3. Reasoning: A Vision Model analyzes the interface and understands the current state.
4. Action: The model outputs tap coordinates or other actions.
5. Execution: The Agent uses ADB (Android Debug Bridge) to execute the actions on the phone.
6. Loop: The process repeats until the task is complete.

Architecture Components:

• Agent Code: Runs on your computer; handles model calls, action parsing, and phone control.
• Vision Model Service: A remote API or a locally deployed service that provides the reasoning capabilities.

Follow these steps to install the agent code on your local machine.

Prerequisites:

• ADB tools installed and accessible via command line (adb version should work).
• Python environment.

Installation Steps:

1. Clone the repository:

   git clone https://github.com/zai-org/Open-AutoGLM.git
   cd Open-AutoGLM

2. Create and activate a virtual environment:

   python -m venv venv
   # On macOS/Linux:
   source venv/bin/activate
   # On Windows:
   venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt
   pip install -e .

Follow these steps to prepare your environment for the Agent:

1. Install ADB Tools

• MacOS: brew install android-platform-tools
• Windows: Download the platform-tools from the Android developer website and add the directory to your PATH.

2. Verify ADB

Run adb version to ensure it is installed, then connect your phone and run adb devices to confirm the connection.

3. Install Agent Dependencies

Clone the repository, create a virtual environment, and install the required packages:

git clone https://github.com/zai-org/Open-AutoGLM.git
cd Open-AutoGLM
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt
pip install -e .

Your device must meet the following requirements:

• Android 7.0+ device or emulator.
• Developer Mode enabled (Tap Build Number in Settings > About Phone 10 times).
• USB Debugging enabled in Settings > Developer Options.
• ADB Keyboard installed: Download ADB Keyboard APK and enable it in Settings > Input Method or Settings > Keyboard List to allow text input via automation.

Crucial for Tapping: If the agent can open apps but cannot tap elements, enable USB Debugging (Security Settings) in Settings > Developer Options.

You can use existing model services like Zhipu BigModel or ModelScope by providing the --base-url, --model, and --apikey flags. This is the recommended approach as it does not require local GPU resources.

Using Zhipu BigModel:

• --base-url: https://open.bigmodel.cn/api/paas/v4
• --model: autoglm-phone

Using ModelScope:

• --base-url: https://api-inference.modelscope.cn/v1
• --model: ZhipuAI/AutoGLM-Phone-9B

Example Command (Zhipu):

python main.py --base-url https://open.bigmodel.cn/api/paas/v4 --model "autoglm-phone" --apikey "your-bigmodel-api-key" "打开美团搜索附近的火锅店"

To install Open-AutoGLM, ensure you have Python 3.10 or higher installed. Follow these steps to install dependencies and the package in editable mode:

1. Install dependencies via pip.
2. Install the package using pip install -e ..

# Install dependencies
pip install -r requirements.txt

# Install package
pip install -e .

Open-AutoGLM requires ADB to communicate with Android devices.

1. Download the official ADB platform-tools.
2. Add the extracted folder to your system PATH.

macOS Example:

# Assuming extracted to ~/Downloads/platform-tools
export PATH=${PATH}:~/Downloads/platform-tools

Windows: Add the extracted folder path to your system environment variables.

# macOS example
export PATH=${PATH}:~/Downloads/platform-tools

To use Open-AutoGLM with iOS devices, you must meet the following requirements:

• Operating System: macOS
• Software: Xcode (latest version from the App Store)
• Account: Apple Developer Account (a free account is sufficient)
• Hardware: iOS device (iPhone/iPad) and a USB data cable (or ensure both Mac and iPhone are on the same WiFi network)

Install the required Python packages to prepare the project for deployment:

pip install -r requirements.txt 
pip install -e .