computer-use-linux

computer-use-linux is a Rust-based Model Context Protocol (MCP) server and CLI designed to provide full control of a Linux desktop to an AI agent. Unlike many Linux computer-use implementations that rely on X11 or OCR, this project is Wayland-first and uses semantic accessibility (AT-SPI) to interact with the desktop.

Key capabilities include:

• Desktop Interaction: Reading accessibility trees, listing/focusing windows, taking screenshots, clicking, dragging, scrolling, typing, and invoking semantic actions.
• Wayland Support: Uses org.freedesktop.portal.RemoteDesktop for pointer actions, wtype for text input (preserving Unicode/layouts), and various DBus methods for screenshots.
• Compositor Awareness: Supports GNOME, KDE/KWin, Hyprland, i3, and COSMIC via specialized window registry backends.
• Semantic Selectors: Allows agents to target UI elements using role, name, text, or states instead of relying solely on pixel coordinates.
• MCP Integration: Can be spawned by any MCP host (e.g., Claude Desktop, Hermes Agent) to gain desktop control.

The computer-use-linux MCP server exposes a suite of tools categorized by their function. These tools allow an agent to diagnose the system, discover the desktop state, perform inputs, and execute semantic actions or shell commands.

Diagnostics

• doctor: Generates a single-shot JSON readiness report (platform, portals, accessibility, windowing, input, etc.).
• setup_accessibility: Enables GNOME's org.gnome.desktop.interface toolkit-accessibility setting.
• setup_window_targeting: Installs and enables the bundled GNOME Shell extension for window control.

Discovery

• list_apps: Lists running desktop apps visible to the AT-SPI registry.
• list_windows: Lists compositor windows with metadata (title, app id, wm_class, focus state, client type, bounds).
• focused_window: Returns the window currently holding keyboard focus.
• get_app_state: Returns a combined screenshot and accessibility tree for a specific app. Use app_name_or_bundle_identifier or a window target to scope it.
• screenshot: Captures the screen as a PNG or JPEG. Can be targeted to a specific window.

Input

• click: Performs a click via element index, semantic selector, or desktop coordinates.
• drag: Performs a coordinate-based drag (start/end).
• scroll: Performs a page-based scroll on an element or at a pixel location.
• press_key: Sends keys or chords; can focus a window or terminal first.
• type_text: Inputs literal text, optionally targeted at a window or terminal.

Semantic Actions & Navigation

• perform_action: Invokes an AT-SPI action (e.g., Press, Activate, Toggle) on an element.
• set_value: Writes to settable elements like text fields, sliders, or spinners.
• activate_window: Focuses a window by window_id, pid, app_id, wm_class, title, or terminal selectors.
• move_window / resize_window: Repositions or resizes a window in desktop coordinates.

Conditional Tools

• run_shell: Executes /bin/sh -c commands. Only available if the server is started with COMPUTER_USE_LINUX_ENABLE_SHELL=1.
• complete_interaction: Sends a desktop completion notification. Only available if COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1 is set.

The project uses a multi-layered approach to interact with Linux desktops:

• Accessibility Tree: Uses the atspi crate to talk to the AT-SPI registry. It flattens the tree into (role, name, text, states, bounds) tuples for indexing.
• DBus Communication: Uses zbus for portal calls (Screenshot, RemoteDesktop, ScreenCast), GNOME Shell screenshots, and the bundled GNOME extension (dev.avifenesh.ComputerUseLinux.WindowControl).
• MCP Transport: Uses rmcp with stdio framing (no network).
• Input Handling:
  • X11: Prefers xdotool/XTEST. If xdotool is unavailable, it falls back to ydotool.
  • Wayland: Uses wtype (supporting Unicode via virtual-keyboard protocol on wlroots/Hyprland) or falls back to ydotool 1.0.3+.
• Window Registry: A unified backend that probes in order: GNOME extension $\rightarrow$ GNOME Introspect $\rightarrow$ COSMIC helper $\rightarrow$ KWin scripting $\rightarrow$ Hyprland hyprctl $\rightarrow$ i3 IPC $\rightarrow$ generic X11/EWMH.
• Terminal Enrichment: list_windows cross-references terminal windows with their controlling TTY and foreground process, allowing tools like type_text to target specific running processes (e.g., "the terminal where pytest is running") without needing a window ID.

The computer-use-linux server is not read-only and can mutate application and desktop state. MCP hosts can use ToolAnnotations to surface safety hints to users before invocation.

Note: Annotations are hints, not an authorization system. Hosts should still prompt users before performing destructive actions.

The integration uses additive dynamic tool loading. Pi initially only sees a single loader tool: computer_use_linux_tools. You must use this loader to enable specific native tools for the session.

Enabling tools

You can enable tools by providing an exact list or a capability query:

By exact tool name:

computer_use_linux_tools({ tools: ["doctor", "list_windows"] })

By capability query:

computer_use_linux_tools({ query: "inspect a window and type text" })

Once enabled, the tools appear in the next model turn with their full schemas (e.g., computer_use_linux_doctor({}), computer_use_linux_list_windows({}), computer_use_linux_type_text({ text: "hello", title: "Notes" })).

Process Lifecycle

Pi does not start the desktop process when the loader runs. The first real tool call starts one computer-use-linux process, which is reused for the duration of the session to ensure get_app_state element indices and portal sessions remain valid.

Because this tool provides computer control, users should be aware of the following security boundaries:

• ydotoold Permissions: ydotoold runs as a per-user service with access to /dev/uinput. Any process able to connect to its socket (default: /run/user/$UID/.ydotool_socket, mode 0600) can synthesize arbitrary input. Do not run ydotoold as root.
• Screencast Portal: Granting permission once per session allows the MCP host to capture the screen for the remainder of that session. If you wish to avoid this, use get_app_state with include_screenshot: false.
• AT-SPI: The AT-SPI bridge (enabled via setup_accessibility) exposes window contents to any client on your session bus. This is the same trust boundary used by screen readers.
• Network Isolation: The binary performs no network operations (no TCP/UDP listeners, no outbound connections, no telemetry). It relies entirely on local transports like DBus and Unix sockets.
• Tool Mutability: The MCP tool list distinguishes between read-only and mutating tools. While these are hints, the host is responsible for enforcing user approval policies.

When using computer-use-linux to control a desktop, follow these operational patterns:

1. Scope State: Always begin a turn with get_app_state. Scope it to a specific app_name_or_bundle_identifier or window target (window_id, pid, app_id, wm_class, title) to avoid flooding context with the entire desktop tree.
2. Verify Windows: Before sending input, call list_windows or focused_window to verify the target window by title, app id, pid, or wm class.
3. Prefer Semantic Targeting: Use element indices or role/name/text/states selectors from the accessibility tree rather than raw coordinates.
4. Targeted Text Input: Prefer type_text with a target selector (e.g., window_id, pid, app_id) rather than relying on current focus.
5. Handle Truncation: If accessibility_tree_truncated is true, do not lower max_nodes or max_depth. Instead, scope to a narrower target (a specific window or element) to get a more detailed local tree.