Appium UiAutomator2 Driver

The driver supports UiScrollable to handle elements that are not currently visible on the screen. You can use UiScrollable to find a scrollable container and then use methods like getChildByText or scrollIntoView to locate specific elements.

• getChildByText: Finds a child element within a scrollable container by its text.
• scrollIntoView: Scrolls the container until the specified UiSelector is visible and returns that element.

# Find a TextView with text "Tabs" inside a scrollable container
element = find_element(:uiautomator, 'new UiScrollable(new UiSelector().scrollable(true).instance(0)).getChildByText(new UiSelector().className("android.widget.TextView"), "Tabs")')

# Scroll to an element matching a UiSelector and return it
element = find_element(:uiautomator, 'new UiScrollable(new UiSelector().scrollable(true).instance(0)).scrollIntoView(new UiSelector().text("WebView").instance(0));')

When testing multi-window applications (Split-screen, PiP, Freeform, System overlays), it is important to distinguish between three layers of abstraction:

1. Windows: Containers for UI elements belonging to an application. Each window has a windowId, packageName, bounds (rect), Z-order (layer), and state flags like isActive or isFocused.
2. Displays (Logical Displays): Virtual screens that contain windows. The default display typically has displayId = 0. Additional displays can be external monitors or virtual displays.
3. Hardware Displays (Physical Displays): The actual physical screen hardware. Multiple logical displays can map to a single physical display.

Multi-Display Support by Android Version:

• Android API 30+ (Android R): Full support. The driver can access windows across all displays via getWindowsOnAllDisplays(). You can target a specific display using the currentDisplayId setting.
• Before Android API 30: Only the default display (displayId = 0) is accessible. currentDisplayId has no effect.

The UiAutomator2 driver provides two distinct ways to stream the screen. Choose based on your requirements:

Device MJPEG Server (Recommended for simple streams)

• Mechanism: Runs entirely on the Android device via the built-in UiAutomator2 server.
• Setup: Requires setting appium:mjpegServerPort to forward the port to the host.
• Pros: No host-side dependencies; simple TCP/HTTP stream.
• Use Case: Live viewing in a browser, simple screen recording (e.g., via ffmpeg), or using the stream for screenshots.

mobile: startScreenStreaming / stopScreenStreaming (GStreamer-based)

• Mechanism: Starts a host-side MJPEG server using GStreamer to capture and broadcast the screen.
• Setup: Requires the adb_screen_streaming feature and GStreamer (with gst-plugins-base, gst-plugins-good, and gst-plugins-bad) installed on the host machine.
• Pros: Advanced encoding and network options via GStreamer pipelines.
• Use Case: Complex host-side streaming requirements.

The Appium UiAutomator2 Driver operates within an automation host to bridge the gap between a test client and an Android device. The architecture follows this flow:

1. Test Client: Test code uses an Appium Client Library (Java, Python, JS, etc.) to send W3C WebDriver commands over HTTP to the Appium Server.
2. Automation Host: The Appium Server forwards session commands to the UiAutomator2 Driver. The driver manages the connection to the device using ADB (Android Debug Bridge) for port forwarding and Chromedriver Management (specifically for hybrid apps or webview contexts).
3. Device Target: The driver communicates with the UiAutomator2 Server (an instrumentation HTTP API) on the device via ADB port forwarding. The UiAutomator2 Server then interacts with the UiAutomator Framework to perform UI interactions and access the accessibility tree of the Application Under Test (AUT). For webview contexts, Chromedriver is used to provide WebDriver capabilities directly within the webview.

flowchart TD
  subgraph ClientSide["Test Client"]
    T["Test Code"]
    CL["Appium Client Library<br/>(Java / Python / JS / Ruby / C#)"]
  end

  subgraph ServerHost["Automation Host"]
    AS["Appium Server<br/>WebDriver HTTP API"]
    XD["UiAutomator2 Driver<br/>(appium-uiautomator2-driver)"]
    ADBM["ADB + Port Forwarding"]
    CDM["Chromedriver Management<br/>(hybrid / webview only)"]
  end

  subgraph DeviceTarget["Android Device / Emulator"]
    U2S["UiAutomator2 Server<br/>(instrumentation HTTP API)"]
    UIA["UiAutomator Framework"]
    CD["Chromedriver<br/>(in webview context)"]
    AUT["Application Under Test"]
  end

  T --> CL
  CL -->|"W3C WebDriver over HTTP"| AS
  AS -->|"Forwards session commands to driver"| XD
  XD -->|"Install, shell, forward ports"| ADBM
  XD -->|"Context switch to WEBVIEW_*"| CDM
  ADBM -->|"adb forward (e.g. host:8200 → device:6790)"| U2S
  CDM -->|"Chromedriver HTTP"| CD
  U2S -->|"UiAutomator APIs"| UIA
  UIA -->|"UI interactions + accessibility tree"| AUT
  CD -->|"WebDriver in webview"| AUT

Scheduled Actions allow you to run code on the server side asynchronously to handle transient UI elements (like popups or notifications) that might disappear before a standard HTTP-based WebDriver command can reach them.

Key Concepts:

• Asynchronous Execution: You describe an action in JSON, and the server executes it on the main UI thread independently of your test script's execution flow.
• Action Types: Currently supports gesture (emulating user input), source (taking XML page source), and screenshot (taking PNG screenshots).
• Lifecycle: Actions are scheduled by the client, run on the server, and can be inspected via history or stopped using unscheduleAction. All scheduled actions are automatically reset when a new session is created.
• Availability: This feature is available in the UiAutomator2 driver since version 2.26.0.

# Example of the mental model: 
# 1. Schedule an action to run in the background
driver.execute_script('mobile: scheduleAction', { ... })

# 2. Perform other test steps that might trigger the UI event
do_other_test_steps()

# 3. Stop the action and retrieve the history to assert what happened
history = driver.execute_script('mobile: unscheduleAction', { 'name': 'myAction' })

UiAutomator2 uses XPath2 by default (via Psychopath). If you encounter issues with sophisticated locators, you can force the driver to use the standard Android XPath1 implementation by setting enforceXPath1 to true.

Additionally, you can control how context-based searches behave using limitXPathContextScope. By default (true), searches are limited to the parent element. Setting this to false allows the search to access the entire page source, but you must use the . notation (e.g., .//element) to ensure you are still searching within the intended descendant scope.

// To fix issues with complex XPath2 locators:
driver.setSetting("enforceXPath1", true);

// To allow context-based searches to see the whole tree:
driver.setSetting("limitXPathContextScope", false);
// Note: When limitXPathContextScope is false, use dot notation for descendants:
// driver.find_element(:xpath, ".//element")