MobileWorld Benchmark

A common failure mode for mobile agents is repeating the same ineffective action (e.g., clicking a non-interactive label or scrolling a list that doesn't move).

To implement effective loop-breaking, the agent should:

1. Detect when an action produces no observable change in the UI.
2. Escalate or switch the action_type to a different interaction pattern (e.g., switching from click to drag, or from scroll to long-press) to break the stall.

# Example of escalating from a failed click to a successful drag

# Failed attempt (Looping)
Thought: I need to click on "Brightness level" to access the brightness slider.
Action: {"action_type": "click", "coordinate": [288, 451]}

# Successful loop-breaking (Escalation)
Thought: I can see the brightness slider at the top of the screen. It's currently at about 48%. I need to drag the slider to the maximum.
Action: {"action_type": "drag", "start_coordinate": [279, 115], "end_coordinate": [540, 115]}

Even with model upgrades (like Claude Opus 4.7), certain task clusters consistently fail due to navigation and resource constraints rather than reasoning gaps. Common failure mechanisms include:

• Advanced UI Navigation (e.g., Mastodon): Agents fail when Android clients lack features available on web versions, or when encountering naming ambiguities (e.g., confusing 'posting language' with 'account language').
• Long-thread Workflows (e.g., Mattermost):
  • Phantom execution: The agent's thought log claims an action (like attaching a file) was successful, but the trace shows the action was never actually executed.
  • Step-budget exhaustion: High working-memory loads (reading 10+ messages) cause the agent to exhaust its 50-step budget before completing the task.
• PDF/Document Reading: Agents often get stuck in search-and-scroll loops, finding text mentions but failing to navigate to specific UI elements like table cells.
• Archive/Filtering Workflows: Agents may exhaust their step budget performing tedious manual filtering (e.g., checking file dates in a list) before reaching the final goal (e.g., zipping and emailing).

The Kimi K2.6 upgrade shows significant improvements over K2.5, primarily through better loop-breaking and environment awareness.

Key improvement patterns include:

• Loop-breaking: K2.6 detects when actions (like scrolling or clicking) fail to change the environment state and pivots to a new strategy, whereas K2.5 often repeats the same action for 30–50 steps.
• Better environment knowledge: K2.6 has improved awareness of app-specific patterns, such as GitHub API endpoints, Mattermost member lists, and Android "Copy to" flows.
• Multi-app workflow completion: K2.6 better manages step budgets when transitioning between apps (e.g., moving from a calendar to an alarm, or email to SMS).

Regression Note: K2.6 exhibits a regression in over-precise semantic interpretation. It may fail keyword-based tasks by being too linguistically precise (e.g., excluding "short-sleeve pajamas" from a search for "short-sleeve T-shirts").

GUI grounding is the ability of a multimodal model to accurately identify and interact with specific pixels or UI elements. There are three primary paradigms used to achieve this:

1. Set-of-Mark (SoM): A specialized parser (like OmniParser V2) identifies UI elements and overlays bounding boxes/identifiers on the interface. The MLLM then acts as a reasoning engine to select the correct identifier. This method often suffers from high visual noise and ambiguity in dense layouts.
2. End-to-End (E2E): The model processes raw screenshots directly and predicts (x, y) coordinates without external parsers. While more direct, performance can drop significantly on high-resolution displays with dense layouts.
3. Inference-Phase Optimization (Interactive): Transforms grounding into a multi-step process using tools like a Zoom-In (crop-and-zoom) tool. This allows the model to inspect high-resolution areas more closely, significantly improving spatial precision.

MobileWorld is a benchmark for autonomous mobile agents using a containerized, deterministic approach.

System Architecture

• Containerized Environment: Runs in Docker-in-Docker, including a rooted Android Virtual Device (AVD), application backends, and an orchestration API server.
• Open-Source Applications: Uses stable backends like Mattermost, Mastodon, and Mall4Uni to allow precise control over task initialization.
• Snapshot-Based State Management: Uses AVD snapshots to ensure every task starts from an identical, reproducible state.

Verification Methods

Tasks are verified using:

• Textual Answer Verification: Pattern matching/string comparison.
• Backend Database Verification: Direct queries to validate state changes (e.g., messages, posts).
• Local Storage Inspection: ADB-based inspection of app data (e.g., calendar, email).
• Application Callbacks: Custom APIs for intermediate state validation.