agent-device

agent-device is a tool designed for agent-native app automation across mobile, TV, and desktop platforms. It provides coding agents with structured UI access, deterministic interactions, and app-level evidence.

Key capabilities include:

• Multi-platform control: One CLI to control iOS, Android, tvOS, Android TV, Amazon Vega OS TV (via Vega Virtual Device), macOS, and Linux desktop targets.
• Structured UI Context: Uses accessibility-first snapshots to provide agents with accessibility trees, reducing the need for screenshot-only reasoning.
• Agent-native Interactions: Supports actions such as tap, pan, fling, pinch, rotate, scroll, focus, type, assert, and finding visible UI via refs, selectors, and semantic finders.
• Debugging & Profiling: Built-in support for collecting logs, inspecting traffic, capturing screens/recordings, and sampling performance.
• Session & Replay: Ability to open apps, maintain stateful context, and replay recorded .ad actions to reproduce flows without requiring AI at runtime.
• React Native Support: Specialized tools including agent-device react-devtools for component trees/render profiles and agent-device cdp for JS heap snapshots, diffs, and leak retainers.

agent-device is an agent-native CLI designed for app verification and QA by coding agents. It provides the 'hands, eyes, and evidence collector' for agents to interact with real applications across multiple platforms including iOS, Android, tvOS, Android TV, Amazon Vega OS TV, macOS, Linux desktop, and a managed web browser surface.

Key capabilities include:

• App verification: Inspecting UI and acting via refs/selectors.
• Token-efficient UI context: Providing structured accessibility snapshots instead of just screenshots.
• Runtime evidence: Capturing screenshots, recordings, logs, network traffic, traces, and performance snapshots (CPU/memory).
• Replayable checks: Creating .ad replay scripts from exploratory sessions.
• React Native/Expo support: Profiling component trees, props, state, and render performance.
• Multi-platform support: Driving simulators, emulators, physical devices, and desktop apps through a single CLI.

The project uses a layered architecture with strict dependency rules enforced by a 'gate' (the TYPE_CYCLE_BASELINE and TYPE_INVERSION_BASELINE in scripts/layering/).

Key architectural constraints:

• Runtime Graph: At runtime, the module graph is a clean Directed Acyclic Graph (DAG).
• Type-Level Cycles: While the runtime is clean, large clusters of files may have type-only dependencies. This is a comprehension obstacle rather than a correctness issue, as types are erased at runtime.
• Layering Rules: Higher-ranked zones (like core or contracts) must not depend on lower-ranked zones (like commands or daemon).
• Composition Roots: Files like runtime.ts, agent-device-client.ts, and provider-device-runtime.ts act as composition roots. They sit outside the main 'spine' because they wire together components from different layers (e.g., wiring commands/ into daemon/).

The engine parses supported Maestro YAML into a source-preserving typed program and executes it through a narrow compatibility runtime port. It avoids lowering the program through SessionAction or recursive replay dispatchers to maintain high fidelity with Maestro's original intent.

Core Responsibilities:

1. Parser: Validates grammar and preserves source path and line numbers for every command.
2. Interpreter: Manages hooks, includes, environment scopes, conditions, and ordered execution. It handles repeat.times (preserving authored counts) and retry (clamping maxRetries to a three-retry preset, totaling four attempts).
3. Runtime Port: Exposes typed operations for app, input, observation, target, and single-pointer gestures.
4. Execution Context: Manages variables and observation generation. Mutations invalidate the current generation, and geometry is local to a specific action.
5. Observer: Adapts traces, artifacts, and failures to existing replay and test result contracts.

The agent-device daemon uses a persistent platform helper session pattern to optimize performance for backends with high startup costs (e.g., iOS XCTest runner or Android UiAutomation). Instead of launching a new process for every command, the daemon maintains a warm session that communicates via a validated protocol. This reduces latency significantly (e.g., reducing Android snapshot p50 from ~260ms to ~16ms).

Key Session Behaviors:

• Lazy Startup: Sessions start on the first command that benefits from reuse.
• Identity Binding: Sessions are bound to a specific device identity and helper/runner identity.
• Automatic Invalidation: Sessions are destroyed and restarted if the device disconnects, the helper version changes, the app identity changes, or a protocol/socket failure occurs.
• Fallback Mechanism: If a persistent session fails, the system falls back to a one-shot execution path for the current command.
• Fail-Closed Policy: If the helper is unavailable or invalid, the system must fail closed rather than substituting with legacy engines (like adb uiautomator dump).

The Unified Request Event Journal is a proposed architecture to consolidate disparate telemetry and diagnostic channels into a single, structured system. It replaces fragmented event vocabularies with a centralized event catalog and a unified journal system.

Key Architectural Principles:

• Event Catalog: A single source of truth (src/contracts/events.ts) where all event kinds (key, subsystem, default level, and derivation traits) are declared. Emitting an undeclared kind results in a compile error.
• Single Journal per Scope: Instead of multiple disparate logs, emitters write to a single journal within a specific scope (e.g., per-request scope or session-scoped teardown scopes).
• Sink-Based Consumption: Consumers are registered as 'sinks' at scope construction. Sinks are invoked synchronously in registration order and operate on a best-effort basis (errors in one sink do not affect others).
• Redaction at the Boundary: Redaction of sensitive data occurs once at the journal boundary for every journal-owned output.
• Separation of Progress Streaming: Progress streaming (src/request/progress.ts) remains a separate, transport-owned channel and is not part of the journal system.

The daemon architecture is designed as a serialized host that manages request scopes rather than acting as a monolithic runtime. The architecture uses a layered approach where a Daemon host manages a Locked request scope, which in turn manages various adapters (e.g., Replay-test daemon adapter, .ad daemon adapter, Maestro daemon adapter).

To maintain security and modularity, the system enforces three explicit visibility levels for its modules:

• Module-public: Internal APIs that other modules may import (but do not add npm exports or change the daemon wire). Imports into an internal/ directory are blocked by boundary gates.
• Repository-private: Leaf contracts used for specific internal purposes (e.g., the canonical .ad codec).
• Daemon-private: Capabilities restricted to locked request and lifecycle coordinators (e.g., SessionScriptPublication, LockedSessionContext).
• Implementation-private: Internal state and logic restricted to the owning engine (e.g., engine IR, parsers).

To use the Agent Device system effectively, understand these key domain concepts:

Session & Device Management

• Session: A daemon-owned state for a selected target and an opened app or surface.
• Target: The automation destination (e.g., mobile, tv, or desktop).
• Modality: The broad device family (e.g., mobile, tv, or desktop).
• Device Lease: Logical remote ownership of a device for a specific tenant/run/client.
• Device Key: A stable, provider-scoped identity (e.g., simulator UDID, physical device ID) used for lease contention.
• Lease Provider: The remote connection source (e.g., proxy, cloud bridge, or limrun) that routes and owns a device lease.

Execution Components

• Interactor: The semantic interface between command dispatch and platform behavior.
• Platform Module: The platform-specific implementation behind the Interactor.
• Runner: The backend process executing commands (e.g., Swift runner for XCTest, TypeScript runner for daemon-side policy).

The agent-device daemon is designed as a serialized application host rather than a container for all runtimes. When building or extending modules, adhere to these core principles:

• Daemon Responsibilities: The daemon manages transport, authentication, request admission/cancellation/locks/leases, request-scoped provider binding, session ownership/teardown, and diagnostics/artifacts.
• Module Responsibilities: Program semantics (e.g., native .ad replay, Maestro, replay-test scheduling) must reside behind independent module APIs.
• Boundary Rules:
  • State must cross boundaries as values.
  • Authority must cross as capabilities.
  • Boundaries exist when a port has two implementations; otherwise, it is considered indirection.
• Dependency Constraints: Do not pass DaemonRequest, DaemonInvokeFn, SessionStore, mutable SessionState, provider handles, or platform implementations directly to an engine. Instead, connect modules to daemon-owned capabilities through adapters created within an admitted and locked request scope.