DOCSEARCHbyAlgolia
Submit Docs

Roubao Documentation

repository·main·Indexed 25 days ago

https://github.com/turbo1123/roubao

An open-source, native Android AI automation assistant that uses Vision Language Models (VLM) to control devices via natural language. Built with Kotlin, Roubao operates on-device using a dual-layer architecture of Tools and Skills, leveraging Shizuku for system-level permissions. It supports multiple VLM providers including Alibaba Tongyi Qianwen, OpenAI, and OpenRouter, and can be configured for local models via Ollama or vLLM.

Tokens
11.5K
Snippets
2
Records
67
Agent score
80%

What's inside Roubao

  1. What is Roubao and how does it work?

    main

    Roubao is an open-source AI Android automation assistant that runs natively on Android devices using Kotlin. Unlike traditional automation tools that require a computer and Python environment, Roubao performs screenshot capture, analysis, and execution locally on the phone.

    Core Architecture: Tools + Skills

    Inspired by Claude Code, Roubao uses a dual-layer architecture to translate natural language into phone actions:

    1. Tools Layer (Atomic Capabilities): Low-level functions that perform single operations, such as search_apps, open_app, deep_link, clipboard, shell, and http.
    2. Skills Layer (User Intent): High-level tasks that map natural language to specific workflows. Skills use two execution modes:
      • Delegation: For high-confidence matches, Roubao uses a deep_link to jump directly to an AI-capable app (e.g., opening Xiaomei AI to order food).
      • GUI Automation: For apps without built-in AI (e.g., WeChat, Meituan), Roubao uses a Vision Language Model (VLM) to perform a "screenshot-analyze-operate" loop to navigate the interface.

  2. How Roubao's Tools and Skills architecture works

    main

    Roubao uses a dual-layer Agent framework inspired by Claude Code, separating low-level capabilities from high-level user intents.

    1. Tools Layer (Atomic Capabilities)

    These are low-level, independent operations used to interact with the system:

    • search_apps: Smart app search (supports pinyin and semantic search).
    • open_app: Opens a specific application.
    • deep_link: Jumps to a specific page within an app using DeepLinks.
    • clipboard: Reads from or writes to the system clipboard.
    • shell: Executes Shell commands.
    • http: Performs HTTP requests to external APIs.

    2. Skills Layer (User Intent)

    Skills map natural language commands to specific execution paths:

    • Delegation (Fast Path): For high-confidence matches, Roubao uses DeepLinks to open AI-capable apps (e.g., Xiaomei AI, Jimeng) to complete the task in one step.
    • GUI Automation (Standard Path): For apps without native AI capabilities (e.g., Meituan, WeChat), Roubao enters an Agent Loop: it takes screenshots, analyzes them via a Vision Language Model (VLM), and performs step-by-step GUI operations (taps, swipes) to complete the task.

  3. Understand the Roubao Architecture

    main

    Roubao uses a multi-layered architecture to transform user intent into device actions. The layers are organized as follows:

    1. UI Layer (Compose): The user interface (HomeScreen, Settings, History).
    2. Skills Layer: Maps user intent to specific workflows (e.g., 'Order Food', 'Navigate'). It uses a SkillManager to perform intent recognition. High-confidence skills can trigger a 'Fast Path' via direct DeepLinks.
    3. Tools Layer: Provides atomic capabilities (e.g., search_apps, open_app, shell, tap, swipe) wrapped by a ToolManager.
    4. Agent Layer: The core intelligence (ported from MobileAgent-v3). It consists of a Manager (Planning), Executor (Execution), Reflector (Reflection), and Notetaker (Notes).
    5. VLM Client: Interfaces with Vision Language Models like Qwen-VL, GPT-4V, or Claude to analyze screen states.
    6. Shizuku/System Layer: Provides the actual system-level control (screencap, input tap, input swipe, am start) required for execution.

  4. How the Roubao architecture works

    main

    Roubao uses a layered architecture to transform user intent into device actions. The hierarchy is as follows:

    1. UI Layer (Compose): The user interface (HomeScreen, Settings, History).
    2. Skills Layer: Maps user intent to specific paths. The SkillManager identifies if a request can be handled by a high-confidence Delegation Skill (direct DeepLink jump) or if it requires the standard Agent loop.
    3. Tools Layer: Provides atomic capabilities via ToolManager. Tools include search_apps, open_app, deep_link, clipboard, shell, http, screenshot, tap, swipe, and type.
    4. Agent Layer: The core intelligence (ported from MobileAgent-v3). It consists of four specialized components:
      • Manager: Planning and state analysis.
      • Executor: Decision making and action execution.
      • Reflector: Evaluating the results of actions.
      • Notetaker: Recording state and progress.
    5. VLM Client: Interfaces with Vision Language Models like Qwen-VL, GPT-4V, or Claude.
    6. Shizuku: Provides system-level control (screencap, input tap/swipe, am start) to execute actions without Root.

    Workflow

    When a user provides an instruction:

    • The system first checks for a matching Skill.
    • High-confidence Delegation Skills trigger a direct DeepLink jump to complete the task immediately.
    • If no direct skill matches, the Standard Agent Loop begins:
      1. Screenshot: Captured via Shizuku screencap.
      2. Manager Planning: VLM analyzes the current state.
      3. Executor Decision: Determines the next step.
      4. Action Execution: Performs tap, swipe, type, or open_app.
      5. Reflector Reflection: Evaluates if the action was successful.
      6. Loop: Repeats until the task is complete or safety limits are reached.

  5. Understand Roubao's safety and Root modes

    main

    Safety Protections

    • Automatic Stop: Roubao automatically halts execution when it detects payment or password entry screens.
    • Visibility: A progress overlay shows exactly what the agent is doing during execution.
    • Manual Control: You can manually stop any task at any time.
    • Encryption: API keys are stored using AES-256-GCM.

    Root Mode

    If Shizuku is running with Root privileges, Roubao can enable Root Mode:

    • Capabilities: Unlocks additional system-level operations.
    • Shell Access: Allows the execution of su -c commands.
    • Detection: The option is automatically detected and enabled only in Root environments; otherwise, it is grayed out.

  6. How the Agent Execution Loop Works

    main

    When a user provides input that does not match a high-confidence 'Delegation Skill' (which would jump directly to a DeepLink), the app enters the Standard Agent Loop:

    1. Screenshot: Captures the current screen using Shizuku screencap.
    2. Manager Planning: The VLM analyzes the screenshot to plan the next step.
    3. Executor Decision: Determines the specific action to take.
    4. Execute Action: Performs the action (e.g., tap, swipe, type, or open_app) using the Tools layer.
    5. Reflector: Evaluates the outcome of the action to see if it succeeded.
    6. Loop: Repeats the process until the task is complete or a safety limit is reached.

  7. Quick Start: Install and set up Roubao

    main

    Roubao is a native Android AI automation assistant that runs locally on your device without a PC. Follow these steps to get started:

    1. Prerequisites

    • Android 8.0 (API 26) or higher.
    • WiFi Connection: Required for Shizuku wireless debugging.
    • Shizuku: Installed and running to provide ADB-level permissions.
    • VLM API Key: A key for a Vision Language Model (e.g., Alibaba Qwen-VL or OpenAI GPT-4V).

    2. Setup Shizuku

    Shizuku allows Roubao to simulate taps and read UI content without Root. Choose one startup method:

    • Wireless Debugging (Android 11+): Enable Wireless Debugging in Developer Options, then select it within the Shizuku app.
    • Computer ADB: Connect via USB and run:
      adb shell sh /storage/emulated/0/Android/data/moe.shizuku.privileged.api/start.sh

    3. Install and Configure Roubao

    1. Download and install the latest Roubao APK from the Releases page.
    2. Open Roubao and authorize it within the Shizuku app.
    3. Crucial: Navigate to Settings in the Roubao app to configure your VLM API Key.

    4. Get an API Key

  9. How to contribute to Roubao

    main

    Contributions via Issues and Pull Requests are welcome. To contribute, follow the standard GitHub workflow:

    1. Fork the repository.
    2. Create a feature branch: git checkout -b feature/amazing-feature
    3. Commit your changes: git commit -m 'Add some amazing feature'
    4. Push to your branch: git push origin feature/amazing-feature
    5. Open a Pull Request on the original repository.
    git checkout -b feature/amazing-feature
git commit -m 'Add some amazing feature'
git push origin feature/amazing-feature

  10. Contribute to Roubao

    main

    You can contribute to the project by submitting Issues or Pull Requests following these steps:

    1. Fork the repository
    2. Create a feature branch: git checkout -b feature/amazing-feature
    3. Commit your changes: git commit -m 'Add some amazing feature'
    4. Push to the branch: git push origin feature/amazing-feature
    5. Open a Pull Request
    git checkout -b feature/amazing-feature
git commit -m 'Add some amazing feature'
git push origin feature/amazing-feature

  11. Install and Configure Roubao

    main

    Prerequisites

    • Android 8.0 (API 26) or higher.
    • WiFi Connection (required for Shizuku wireless debugging).
    • Shizuku installed and running.
    • VLM API Key (e.g., Alibaba Tongyi Qianwen, OpenAI, or Claude).

    Installation Steps

    1. Install Shizuku and start it using the methods described in the Shizuku setup guide.
    2. Install Roubao: Download and install the latest APK from the Releases page.
    3. Authorize: Open the Roubao app and grant it permission within the Shizuku app.
    4. Configure API Key: Crucial step. Navigate to the Settings page within Roubao and enter your VLM API Key.

    Getting an API Key

    • Alibaba Tongyi Qianwen (Recommended for users in China): Visit the Alibaba Bailian Platform, enable DashScope service, and create a key in API-KEY management.
    • OpenAI: Visit the OpenAI Platform and create an API key (requires a proxy).

  12. How to set up Shizuku for Roubao

    main

    Roubao requires Shizuku to obtain system-level permissions (like simulating clicks and reading screen content) without requiring Root.

    1. Install Shizuku

    Download Shizuku from Google Play or GitHub Releases.

    2. Start Shizuku

    You must start the Shizuku service using one of the following methods:

    **Method A: Wireless Debugging (Recommended for Android 11+) **

    1. Go to Settings > Developer Options > Wireless Debugging.
    2. Enable Wireless Debugging.
    3. In the Shizuku app, select the "Wireless Debugging" method to start.

    Method B: Computer ADB

    1. Connect your phone to a computer and enable USB debugging.
    2. Run the following command:
    abd shell sh /storage/emulated/0/Android/data/moe.shizuku.privileged.api/start.sh