Midscene.js

Midscene is an AI-driven automation tool that allows you to automate browser actions, extract data, and perform assertions using multimodal models. It provides multiple ways to interact with web interfaces:

• JavaScript SDK: For programmatic automation within your code.
• Chrome Extension: For interactive, browser-based automation.
• YAML Scripting: For defining automation workflows in a declarative format.

For more detailed information, visit the official website at https://midscenejs.com/.

Midscene.js is an open-source, vision-driven UI automation and testing framework. Unlike traditional automation tools that rely on fragile DOM selectors or accessibility trees, Midscene uses multimodal models to interact with interfaces based on screenshots and natural language descriptions.

Key Benefits

• Low Maintenance: No selectors to update when the UI changes.
• Universal Reach: Can target any element a human can see, including <canvas>, native apps, and cross-origin iframes.
• Visual Assertions: Verify layout, colors, and rendered state rather than just DOM existence.
• Flexible Integration: Use it as a plugin for Playwright / Vitest, or use Midscene Skills to let AI agents act autonomously.

Midscene is an open-source SDK designed for vision-driven UI testing and automation. Unlike traditional tools that rely on the DOM or accessibility trees, Midscene operates using screenshots and multimodal models. This allows it to automate interfaces that are often invisible to structure-based tools, such as <canvas> surfaces, native mobile apps (Android, iOS, HarmonyOS), desktop applications, and cross-origin iframes.

Key capabilities include:

• Resilience to refactors: Since it uses visual recognition rather than CSS/XPath selectors, tests don't break when markup or styles change.
• Visual assertions: You can verify colors, layouts, and rendered states directly.
• Broad platform support: One API covers web, mobile, and desktop.
• Visual reporting: Every run generates a visual report for step-by-step replay.

The Midscene Android Playground is a specialized tool designed for testing and interacting with the @midscene/android CLI. It provides a playground environment to experiment with Android automation capabilities driven by Midscene's multimodal models.

For comprehensive documentation and core concepts, visit the official website at https://midscenejs.com/.

The MobileWorld benchmark evaluates Midscene's performance on Android Agent tasks. The benchmark consists of 117 tasks covering cross-app workflows, long-horizon tasks, user interaction, and tool-augmented tasks in realistic mobile scenarios.

Key metrics for the reported version include:

• Pass@1 score: 78.63% (92/117 tasks passed).
• Benchmark Environment: Uses MobileWorld with a Midscene benchmark adapter that delegates execution via Midscene RPC.

@midscene/web allows you to automate UI actions, extract data, and perform assertions in web browsers using AI. It is designed to work with existing browser automation frameworks rather than replacing them.

To use Midscene for web automation, you can integrate it with one of the following frameworks:

• Playwright: For deep integration within Playwright tests.
• Puppeteer: For automation using the Puppeteer API.
• Bridge mode: For scenarios where you need to connect Midscene to a running browser instance or a different environment.

Midscene is evaluated using the AndroidWorld benchmark from Google Research. This benchmark runs on a live Android emulator and evaluates agents on 116 programmatic tasks across 20 real-world Android apps.

In the reported run, Midscene achieved the following success rates:

• Pass@1: 93.10%
• Pass@2: 95.69%
• Pass@3: 97.41%

Run Configuration Details

• Model Name: Gemini-3.5-Flash
• Midscene version: 1.9.5
• DeepThink: on
• MIDSCENE_REPLANNING_CYCLE_LIMIT: 120