DOCSEARCHbyAlgolia
Submit Docs

usecomputer

repository·main·Indexed 43 days ago

https://github.com/remorses/usecomputer

A fast computer automation CLI and TypeScript library for AI agents. It provides low-level access to mouse, keyboard, and screen capture capabilities, optimized for LLM-based agents like OpenAI and Anthropic. Features include coordinate mapping for scaled screenshots, window-scoped captures, and the ability to observe global input events via SSE. Supports macOS, Linux, and Windows.

Tokens
15.3K
Snippets
51
Records
61
Agent score
54%

What's inside usecomputer

  1. How window-scoped screenshots work

    main

    Window-scoped screenshots capture only a specific application window. This reduces noise (no dock or menu bars) and improves model accuracy.

    Even though the screenshot only shows the window, the coordMap includes the window's position on the desktop. This allows you to use coordinates relative to the screenshot while the CLI automatically maps them to the correct global desktop coordinates.

    Workflow:

    1. List windows to find the target ID: usecomputer window list --json.
    2. Capture the window: usecomputer screenshot ./tmp/app.png --window <WINDOW_ID> --json.
    3. Click within that window using the emitted coordMap: usecomputer click -x 400 -y 220 --coord-map "<COORD_MAP_FROM_STEP_2>".
    # 1. find the window ID
usecomputer window list --json

# 2. screenshot that window
usecomputer screenshot ./tmp/app.png --window 12345 --json

# 3. click using the coord-map
usecomputer click -x 400 -y 220 --coord-map "200,100,1200,800,1568,1045"
    usecomputer window list --json
usecomputer screenshot ./tmp/app.png --window 12345 --json
usecomputer click -x 400 -y 220 --coord-map "200,100,1200,800,1568,1045"

  2. Enable agent-friendly screenshots via Kitty Graphics Protocol

    main

    When the AGENT_GRAPHICS environment variable is set to kitty, the screenshot command emits PNG images inline to stdout using the Kitty Graphics Protocol. This allows AI agents to receive screenshots directly in their context without separate file reads.

    To use this with an OpenCode agent, add the kitty-graphics-agent plugin to your opencode.json:

    {
  "plugin": ["kitty-graphics-agent"]
}

    When using this protocol, the JSON output includes "agentGraphics": true to indicate the image was emitted inline.

  3. Understanding the coord-map

    main

    Screenshots are scaled (longest edge $\le$ 1568px). To ensure clicks land on the correct desktop coordinates, you must pass the --coord-map emitted by the screenshot command to any pointer command (click, hover, drag, mouse move).

    The coordMap consists of 6 comma-separated values: captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight

    • captureX, captureY: Top-left corner of the captured region in desktop coordinates.
    • captureWidth, captureHeight: Size of the captured region in desktop pixels.
    • imageWidth, imageHeight: The actual dimensions of the output PNG after scaling.

  4. How screenshot scaling and coord-maps work

    main

    To keep screenshots model-friendly, usecomputer screenshot scales the output image so the longest edge is at most 1568 px.

    Because the screenshot is a scaled version of the desktop, you must use the coordMap emitted in the screenshot's JSON output to map coordinates from the screenshot back to the real screen. The coordMap follows the format: captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight.

    Workflow:

    1. Run usecomputer screenshot ./shot.png --json to get the image and the coordMap.
    2. Use the coordMap value in subsequent pointer commands (like click or mouse move) using the --coord-map flag.
    # Capture screenshot
usecomputer screenshot ./shot.png --json

# Click using the coord-map from the previous step
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"
    usecomputer screenshot ./shot.png --json
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"
usecomputer mouse move -x 100 -y 80 --coord-map "0,0,1600,900,1568,882"

  5. How the usecomputer feedback loop works

    main

    Automating a desktop follows a continuous feedback loop. Because the UI changes after every interaction (menus opening, pages scrolling), you must never reuse a stale screenshot.

    The Loop:

    1. Screenshot: Capture the current state using usecomputer screenshot --json. The --json flag is mandatory to retrieve the coordMap.
    2. Analyze: Send the screenshot image to the AI model.
    3. Act: The model returns an action (e.g., click at x=740 y=320).
    4. Execute: Run the command (e.g., usecomputer click) using the exact --coord-map provided in the screenshot output from step 1.
    5. Repeat: Take a fresh screenshot to observe the result and continue.
    # 1. take screenshot (always use --json to get coordMap)
usecomputer screenshot ./tmp/screen.png --json
# output: {"path":"./tmp/screen.png","coordMap":"0,0,3440,1440,1568,657",...}

# 2. send ./tmp/screen.png to the model
# 3. model says: "click the Save button at x=740 y=320"

# 4. click using the coord-map from the screenshot output
usecomputer click -x 740 -y 320 --coord-map "0,0,3440,1440,1568,657"

# 5. take a fresh screenshot to see what happened
usecomputer screenshot ./tmp/screen.png --json

  6. Quick start with usecomputer CLI

    main

    Common CLI commands for basic automation:

    # Get current mouse position in JSON format
usecomputer mouse position --json

# Move mouse to specific coordinates
usecomputer mouse move -x 500 -y 500

# Click at coordinates with specific button and count
usecomputer click -x 500 -y 500 --button left --count 1

# Type text
usecomputer type "hello"

# Press a key combination (shortcut)
usecomputer press "cmd+s"
    usecomputer mouse position --json
usecomputer mouse move -x 500 -y 500
usecomputer click -x 500 -y 500 --button left --count 1
usecomputer type "hello"
usecomputer press "cmd+s"

  7. Install usecomputer

    main

    Install the usecomputer desktop automation CLI globally via npm.

    Requirements:

    • macOS: Enable Accessibility permission for your terminal application in System Settings > Privacy & Security > Accessibility.
    • Linux: Requires an X11 session with DISPLAY set (Wayland via XWayland is supported).
    • Windows: Must be run in an interactive desktop session.
    npm install -g usecomputer

  8. Capture window-scoped screenshots

    main

    For better model accuracy, capture only a specific application window instead of the full desktop. This reduces noise and provides a focused image.

    1. Find the window ID: usecomputer window list --json

    2. Screenshot the window: usecomputer screenshot ./tmp/app.png --window <WINDOW_ID> --json

    3. Act using the window's coord-map: When you use the --coord-map from a window-scoped screenshot, usecomputer automatically translates the window-relative coordinates back to the correct desktop position.

    # 1. find the window ID
usecomputer window list --json

# 2. screenshot that window
usecomputer screenshot ./tmp/app.png --window 12345 --json

# 3. act using the coord-map
usecomputer click -x 400 -y 220 --coord-map "200,100,1200,800,1568,1045"

  9. System Requirements for usecomputer

    main

    Ensure your environment meets the following requirements based on your OS:

    • macOS: Accessibility permissions must be enabled for your terminal application.
    • Linux: Requires an X11 session with the DISPLAY environment variable set (Wayland is supported via XWayland).
    • Windows: Must be run in an interactive desktop session. Automation input is blocked on locked desktops.

  11. Install the usecomputer skill for AI agents

    main

    If you are using an AI coding agent (such as OpenCode or Claude Code), install the dedicated skill so the agent understands the screenshot → act → screenshot feedback loop and how to use coordinate mapping correctly.

    npx skills add remorses/usecomputer

  12. Use the usecomputer CLI for desktop automation

    main

    The usecomputer CLI provides a standalone way to perform desktop automation tasks like taking screenshots, controlling the mouse and keyboard, and listing windows/displays without requiring Node.js. It is a native implementation that calls the same underlying functions as the N-API module.

    Common use cases include:

    • Capturing screenshots of specific regions, windows, or displays.
    • Simulating mouse clicks, drags, hovers, and movements.
    • Typing text or pressing specific key combinations.
    • Listing active windows, displays, and desktops.
    • Observing global input events via SSE.