DOCSEARCHbyAlgolia
Submit Docs

Cua

repository·main·Indexed 10 days ago

https://github.com/trycua/cua

A platform for scaling computer-use agents, featuring open-source drivers for background desktop interaction, cross-OS sandboxes, and benchmarking environments for training and evaluation. Includes CUA-Bench for UI automation benchmarking and Lume for unlocking Metal capabilities in macOS guests.

Tokens
472.3K
Snippets
1.3K
Records
2K
Agent score
95%

What's inside Cua

  1. Overview of Cua Reference Material

    main

    Cua's technical specifications are divided into several subsystems. Depending on your goal, you should refer to the following documentation areas:

    • cua-driver: For CLI commands, the MCP tool reference for the stdio MCP server, and behavioral limits for background automation.
    • Lume: For CLI commands, the local HTTP API, and host limits related to managing macOS and Linux VMs on Apple Silicon Macs.
    • Sandbox SDK: For the Python API used to build images, create sandboxes, and drive their interfaces.
    • Cua-Bench: For CLI command groups and the Python task-definition contract used in benchmarking and RL environments.
    • Docs & Code MCP: For information regarding the hosted read-only search over Cua documentation and versioned source code.

  2. Overview of CUA-Bench Basic tasks

    main

    CUA-Bench Basic provides a suite of computer-use interaction tasks designed to benchmark fundamental UI automation capabilities. The tasks are categorized by interaction type:

    Button & Form Interactions

    • click-button: Click specific buttons on a page.
    • fill-form: Fill out and submit forms with various input types.

    Text & Input

    • typing-input: Type text into input fields.
    • spreadsheet-cell: Enter data into spreadsheet cells.

    Selection & Pickers

    • select-dropdown: Select items from dropdown menus.
    • color-picker: Select specific colors from a palette.
    • date-picker: Select dates using a date picker.
    • click-icon: Click specific icons from a grid.

    Controls & Widgets

    • drag-slider: Adjust slider controls to target values.
    • toggle-switch: Enable/disable settings with toggle switches.
    • video-player: Control video playback (play, pause, volume, mute).

    Advanced Interactions

    • right-click-menu: Right-click and select context menu options.
    • drag-drop: Drag and drop items into categories.

  3. Overview of cua-driver Rust crates

    main

    The cua-driver workspace contains several specialized crates:

    CratePurpose
    cua-driverMain CLI/MCP daemon and integration tests
    cua-driver-coreShared protocol, tool models, config, and CDP helpers
    platform-macosmacOS AX, capture, input, browser, and daemon support
    platform-windowsWindows UIA, capture, input, overlay, and diagnostics
    platform-linuxLinux AT-SPI, X11/Wayland, capture, and input support
    cua-driver-testkitTest-only helpers for spawning the daemon and parsing responses
    cua-driver-uiaWindows UIAccess worker
    cursor-overlayCross-platform semantic cursor renderer and bounded compiled-theme loader
    cursor-theme-cliShort-lived dotLottie validation, compilation, preview, and installation sidecar
    pip-previewPackaging preview helper

  4. Overview of NeurIPS 2025 Computer-Use Agent Research

    main

    The NeurIPS 2025 proceedings highlight critical advancements in the Computer-Use Agent (CUA) space, specifically focusing on four key pillars:

    1. Maturing Benchmarks: Evaluations are moving from toy problems to complex environments like macOS (macOSWorld), professional software simulations (TheAgentCompany), and real-world web environments (REAL).
    2. Safety and Security: Research is increasingly documenting how agents fail under adversarial inputs, prompt injections (WASP), and privacy/misuse scenarios (OS-Harm, RiOSWorld).
    3. Grounding Bottlenecks: Significant work is being done to improve the mapping of natural language to specific UI actions (GUI-Actor, SE-GUI, OSWorld-G).
    4. Open-Source Progress: Frameworks like OpenCUA are demonstrating that open-source models (e.g., OpenCUA-72B) can compete with proprietary systems in task success rates.

  5. Overview of Cua Agent

    main
    Cua Agent is a computer-use framework designed to run agentic workflows within sandboxed environments on macOS, Windows, and Linux. It features integrated support for liteLLM, allowing developers to leverage various LLM providers for driving agent actions within these sandboxes.

  6. Overview of Computer Agent Benchmarks

    main

    The Computer Agent Benchmarks system evaluates agent providers in the Computer Agent SDK by testing their GUI grounding capabilities, specifically focusing on click prediction accuracy.

    It supports two types of evaluators:

    1. Computer Agent SDK providers: Models identified by strings (e.g., "huggingface-local/HelloKKMe/GTA1-7B").
    2. Reference agent implementations: Custom model classes that implement the ModelProtocol.

  7. Overview of Cursor Gallery features

    main

    The Cursor Gallery includes an interactive production cursor configurator and displays various animation states.

    Configurable options include:

    • Actions: All twelve actions.
    • Delivery: Optional background or foreground delivery.
    • Targets: Optional ax, pixel, browser, or desktop targets.

    Gallery Content:

    • All fifteen badge context states.
    • Twelve isolated theme-owned action animations.
    • Delivery and target glyphs (displayed only in their authoritative runtime location inside the badge).

  8. Overview of CUA Jukebox

    main

    CUA Jukebox is a coordinated multi-cursor computer-use demonstration that turns MIDI files into a musical performance using background agent cursors.

    Each track in a MIDI file is assigned its own cua-driver session with a uniquely colored agent cursor. As the song plays, these cursors glide to specific widgets (drum pads or melodic pitch strips) and click them in the background to trigger sounds. This creates a polyphonic orchestra driven entirely by background computer-use.

    Key Components:

    • Transport (Controller): The foreground window where the human user clicks PLAY or STOP.
    • Drum Windows (Kick/Snare/Hat): Large pads where any click triggers a fixed drum voice.
    • Melodic Windows (Bass/Lead/Pad/Arp/etc.): Pitch strips where the X-coordinate of the click selects the semitone.
    • Cursor Pools: To handle fast notes or chords, each track maintains a pool of up to 6 identically-colored cursors that actuate concurrently.

  9. Overview of cua-auto

    main
    cua-auto is a lightweight, cross-platform automation library designed for synchronous operations. It provides a pyautogui-style API for controlling mouse, keyboard, screen, window, clipboard, and shell functions. It is compatible with Windows, macOS, and Linux, utilizing pynput for input control and pywinctl for window management.

  10. Overview of Cua TypeScript packages

    main

    The Cua TypeScript libraries are organized into two primary packages:

    @trycua/core

    Provides foundational functionality, including:

    • Telemetry integration with PostHog.
    • Common utilities and shared types.

    @trycua/computer

    An SDK for computer interaction and VM management, providing:

    • A VM provider system (Cloud).
    • OS-specific interaction interfaces.
    • Control over screenshots, keyboard, and mouse.
    • Command execution capabilities.

  11. Overview of Cyclops UniFFI SDK bindings

    main

    The Cyclops SDK provides native bindings for multiple languages. The canonical implementation is in Rust (cyclops-sdk). The authoritative, deterministic generation pipeline produces bindings for Python, Kotlin, Swift, and Ruby.

    Other targets exist but follow different patterns:

    • Go (go-uniffi) and Node.js (ts-uniffi): These are compatibility snapshots that retain direct record constructors and do not use the builder pattern.
    • Browser/WASM (ts-uniffi-browser): This is a generated builder target that regenerates from Rust metadata during its build process.

  12. Overview of cua-driver for GUI automation

    main

    The cua-driver is a tool used to drive native GUI applications on macOS, Windows, and Linux. It allows an agent to snapshot an application's accessibility tree and interact with it using element tokens, native menu paths, exact window geometry, or pixel coordinates.

    Key Capabilities:

    • Snapshotting accessibility trees (AX, UIA, AT-SPI).
    • Acting on snapshot-bound element tokens.
    • Navigating native application menus.
    • Operating via exact window geometry or pixel coordinates.
    • Verifying actions from a fresh state.

    When to use it: Use cua-driver when a task requires operating a native application's UI or window state. If a task can be accomplished via a headless API, CLI, or filesystem operation, prefer those over simulating GUI interactions.