appium-mcp

repository·main·Indexed 19 days ago

https://github.com/appium/appium-mcp

An MCP (Model Context Protocol) server that enables AI assistants to perform mobile automation on Android and iOS. It supports natural language-driven testing, automated test generation, and AI vision for element finding. The server operates in two modes: embedded local drivers using bundled appium-uiautomator2-driver or appium-xcuitest-driver, and remote WebDriver/Appium server mode. Key features include screen recording, OpenTelemetry tracing, and a NO_UI mode for CI/CD performance optimization.

Tokens
36K
Snippets
97
Records
142
Agent score
62%

What's inside appium-mcp

  1. How AI-driven element finding works in appium_find_element

    main

    The appium_find_element tool has been enhanced to support two distinct modes of operation via the strategy field:

    1. Traditional Mode: Uses standard strategy and selector parameters to return a standard elementUUID.
    2. AI Mode: Uses the ai_instruction strategy. Instead of a selector, you provide a natural language instruction. The system captures a screenshot, processes it via a vision model, determines the coordinates, and executes actions (like clicks) using the W3C Actions API (performActions) for cross-platform compatibility (Android/iOS).

    This design allows for seamless integration of natural language commands while maintaining backward compatibility with existing automation scripts.

    ┌─────────────────────────────────────────────────────────┐
    │  appium_find_element (增强的MCP工具)                     
    │  ├─ 传统模式: strategy + selector → elementUUID         
    │  └─ AI模式: ai_instruction → 坐标 → 点击操作            
    └─────────────────────────────────────────────────────────┘
  2. Understand the AI-generated element UUID format

    main

    When using appium_find_element with AI instructions, the returned elementUUID follows a specific coordinate-based format. This UUID is used in subsequent commands like appium_click.

    Format: ai-element:x,y:x1,y1,x2,y2

    • x,y: The center point of the element.
    • x1,y1,x2,y2: The bounding box (bbox) coordinates of the element.
    ai-element:540,156:42,130,1038,182
               ↑   ↑   ↑   ↑   ↑    ↑
               x   y   x1  y1  x2   y2
               (中心点)  (边界框 bbox)
  3. How AI Vision element finding works

    main

    AI Vision finding allows users to locate mobile elements using natural language descriptions (e.g., "the search button in the top right") instead of traditional XPath or IDs.

    The workflow is as follows:

    1. The user provides a natural language description.
    2. appium_find_element is called with strategy=ai_instruction.
    3. The system captures a screenshot of the current mobile screen.
    4. The image is compressed and sent to a Vision Large Language Model (VLM) API.
    5. The VLM returns the bounding box (bbox) coordinates.
    6. The system returns a UUID in the format ai-element:x,y:x1,y1,x2,y2.
    7. appium_click is called using the returned elementUUID to perform a click via the W3C Actions API at those coordinates.
  4. MCP Tool Design Principles

    main

    Tools in MCP Appium are designed for consumption by LLMs via the Model Context Protocol. To ensure high reliability in tool selection and invocation, follow these three principles:

    1. One user intent = one tool: Map a tool to a single, coherent goal (e.g., "control the device screen" or "manage an app's lifecycle"). Avoid bundling unrelated intents into one tool.
    2. Minimal, predictable parameter surface: Keep parameters tight. Aim for 3–5 required fields and a limited set of optionals. If you find yourself adding more than 3 conditional fields (fields that are only valid under a specific action), consider splitting the tool.
    3. Errors must teach the LLM what to do next: Error messages should be actionable instructions for the LLM to recover (e.g., suggesting a different action or a valid sessionId) rather than just stating a failure.
  5. Implement the Tool Response Contract

    main

    All new tools must follow a specific response contract to distinguish between successful execution and tool-execution errors. This allows the LLM to differentiate between a failed task and a protocol error.

    Response Rules

    SituationWhat to return
    Success{ content: [{ type: 'text', text: '...' }] } (do not include isError)
    Tool-execution error (e.g., no session, wrong platform, bad input, Appium failure){ content: [{ type: 'text', text: '...' }], isError: true }
    Unknown tool / malformed requestHandled automatically by the MCP protocol layer; do nothing.

    CRITICAL: Never throw for tool-execution errors. Throwing causes the MCP layer to prefix the message with "Tool 'xxx' execution failed: ", which interferes with the LLM's ability to parse the actual cause. Always return the error explicitly using errorResult.

    // Success example
    return textResult(`Action completed: ${JSON.stringify(raw)}`);
    
    // Error example
    return errorResult(`Failed to perform action. ${toolErrorMessage(err)}`);
  6. Use Tool Annotations for AI Guidance

    main

    Annotations help the AI model understand the nature of the tool and when it is appropriate to call it:

    • readOnlyHint: true: Use when the tool only retrieves/reads data without modifying device or session state.
    • readOnlyHint: false: Use when the tool performs actions, modifications, or state changes.
    • openWorldHint: true: Use when the tool requires knowledge or interaction with systems outside the mobile device.
    • openWorldHint: false: Use for operations strictly contained within the device/codebase context.
  7. Understand the AI Finding UUID Format

    main

    When appium_find_element finds an element using AI, it returns a specific UUID format. This string contains the center coordinates and the bounding box of the element:

    ai-element:{x},{y}:{x1},{y1}:{x2},{y2}

    • {x},{y}: The center coordinates of the element.
    • {x1},{y1}: The top-left corner of the bounding box.
    • {x2},{y2}: The bottom-right corner of the bounding box.
    ai-element:540,156:42,130,1038,182
               ↑   ↑   ↑   ↑   ↑    ↑
               x   y   x1  y1  x2   y2
            (center)  (bounding box)
  8. Use MCP Appium in non-English languages

    main
    The AI assistant within MCP Appium is multilingual. You can provide instructions, descriptions, and automation requests in your native language (e.g., Spanish, Chinese, Japanese, Korean, French, German), and the AI will process the request and generate the appropriate automation logic.
  9. Extend Appium MCP with Plugins

    main

    You can compose the default Appium MCP server with custom business logic using the appium-mcp/core package. This allows you to register custom MCP tools, prompts, resources, and resource templates without maintaining a fork of the main repository.

    Key Concepts

    • Registration: Plugins use the McpRegistry to add capabilities. Tools, prompts, and resources follow the FastMCP definition shapes.
    • Lifecycle Hooks: Plugins can wrap tool execution with beforeCall and afterCall hooks. Note that these hooks only apply to tools, not to prompts or resources.
    • Discovery Control: Use the policy option in createAppiumMcpServer to hide specific tools or resources from MCP discovery using regular expressions.
    • Naming Collisions:
      • Plugin Names: Must be unique. If two plugins share a name, the first one wins and the second is skipped with a warning. Use prefixed names like acme-checkout-plugin to avoid collisions.
      • Tool Names: Follow FastMCP behavior (last-registration-wins). Since Appium MCP registers built-in tools first, a plugin tool with the same name will override a built-in tool.
    import { createAppiumMcpServer } from 'appium-mcp/core';
    import type { AppiumMcpPlugin, McpRegistry, ToolCallContext } from 'appium-mcp/core';
    import { z } from 'zod';
    
    class CheckoutPlugin implements AppiumMcpPlugin {
      readonly name = 'checkout-plugin';
      readonly version = '1.0.0';
    
      register(registry: McpRegistry): void {
        const parameters = z.object({ orderId: z.string() });
        registry.addTool({
          name: 'assert_checkout_summary',
          description: 'Assert that the checkout summary screen shows an expected order ID.',
          parameters,
          execute: async (args) => {
            const { orderId } = parameters.parse(args);
            return {
              content: [{ type: 'text', text: `Assert checkout order ${orderId}` }],
            };
          },
        });
      }
    
      async beforeCall(ctx: ToolCallContext): Promise<void> {
        if (ctx.toolName === 'appium_gesture') {
          console.error(`[checkout-plugin] about to call ${ctx.toolName}`);
        }
      }
    }
    
    const server = await createAppiumMcpServer({
      plugins: [new CheckoutPlugin()],
      additionalInstructions: 'Custom checkout policies are active.',
      policy: {
        allowTools: [/^appium_session_management$/, /^assert_checkout_summary$/],
        allowResources: [/^Generate Code With Locators$/],
      },
    });
    
    await server.start({ transportType: 'stdio' });
  10. Decide when to create a new tool vs. consolidate

    main

    Use the following rubric to decide if multiple actions should be grouped into a single tool or kept as separate tools.

    Consolidate into one tool if:

    • The actions serve a single user intent (e.g., "manage app lifecycle").
    • The actions share most parameters (e.g., id, name, sessionId).
    • Adding the action adds ≤3 new conditional parameters (fields valid only under a specific action).
    • The total number of actions in the tool stays ≤10.

    Keep tools separate if:

    • The actions reflect different intents (e.g., "mutate device state" vs "query device state").
    • The actions have disjoint parameter sets.
    • Consolidation would push the tool past ~15 total fields or ~10 actions.
    • The actions occur at different points in a test flow (e.g., configuration vs runtime).
  11. How AI-powered element finding works in appium_find_element

    main

    The appium_find_element tool has been enhanced to support both traditional locator strategies and AI-based element location.

    When using AI mode, you provide a natural language instruction via the ai_instruction strategy. The tool follows this workflow:

    1. Captures and compresses a screenshot of the current screen.
    2. Sends the image and your instruction to a configured vision model API.
    3. Parses the model's bounding box (BBox) response and converts it into coordinates.
    4. Executes a tap action using the W3C Actions API (performActions) to interact with the element.

    This approach allows you to find elements using descriptions (e.g., "the login button") instead of strict selectors like XPath or ID, while maintaining backward compatibility with traditional methods.

    appium_find_element (Enhanced MCP Tool)
      ├─ Traditional: strategy + selector → elementUUID
      └─ AI Mode: ai_instruction → coordinates → tap action
  12. Efficiently find and interact with elements

    main

    When automating element interaction, follow this priority order for optimal performance and reliability:

    1. appium_get_active_element (Priority 1): Use this to retrieve the currently focused element. It is lightweight and returns a single element UUID.
    2. appium_find_element (Priority 2): Use this to target specific elements. When using this tool, follow this selector strategy priority:
      • accessibility id (Highest priority)
      • id
      • Platform-native selectors (-ios predicate string / -ios class chain for iOS, -android uiautomator for Android)
      • xpath (Use only as a last resort)
    3. generate_locators (Priority 3): Use this for debugging or inspecting page structure. It parses the entire page source and returns locators for all interactable elements.