DOCSEARCHbyAlgolia
Submit Docs

Ghost OS Documentation

repository·main·Indexed 23 days ago

https://github.com/ghostwright/ghost-os

Ghost OS is an MCP server written in Swift that enables AI agents to see and operate native macOS applications. It utilizes the macOS accessibility tree (AX Tree) and a local ShowUI-2B vision model for visual grounding. The toolset provides 29 MCP tools for perception, interaction, and window management, and features a self-learning recipe system to record and automate complex multi-step workflows.

Tokens
6.3K
Snippets
12
Records
31
Agent score
82%

What's inside Ghost OS

  1. Understand focus requirements for Ghost OS tools

    main

    Ghost OS tools are categorized by whether they require the target application to be in focus:

    Background Tools (No focus needed)

    These tools work regardless of whether the app is focused or even in a different Space. Note that ghost_screenshot will fail if all windows of an app are closed.

    • ghost_context, ghost_state, ghost_find, ghost_read, ghost_inspect, ghost_element_at, ghost_screenshot

    AX-Native Tools (No focus needed, but may auto-focus)

    These attempt AX-native interaction first, falling back to synthetic events that auto-focus the app.

    • ghost_click, ghost_type

    Focus-Required Tools (Always pass the app parameter)

    These tools require the target app to be focused to function correctly.

    • ghost_press, ghost_hotkey, ghost_scroll, ghost_hover, ghost_long_press, ghost_drag

  2. How Ghost OS works with AI agents

    main

    Ghost OS provides AI agents with "eyes and hands" on macOS by connecting via the Model Context Protocol (MCP).

    Core Mechanisms

    • Accessibility Tree (AX Tree): The primary way Ghost OS "sees" is by reading the macOS accessibility tree, which provides structured, labeled data about UI elements. This is more reliable than screenshot-based guessing.
    • Vision Fallback: For web apps or dynamic content where the AX tree is insufficient, Ghost OS uses a local vision model (ShowUI-2B) to perform visual grounding.
    • MCP Integration: It exposes 29 tools to any MCP-compatible client (e.g., Claude Code, Cursor, VS Code) to allow the agent to interact with any native macOS app (Slack, Finder, Messages, etc.).

  3. Learn and save workflows with self-learning recipes

    main

    Ghost OS allows you to teach the system new workflows by recording your manual actions. A frontier model (like Claude) can synthesize these observations into a parameterized, JSON-based recipe that a smaller, faster model can execute repeatedly.

    Learning Workflow

    1. Start Learning: Call ghost_learn_start with a task_description.
    2. Perform Task: The user manually performs the clicks, typing, and app switching.
    3. Stop Learning: Call ghost_learn_stop to return the enriched action sequence.
    4. Save Recipe: Use ghost_recipe_save to store the synthesized recipe.

    Running a Recipe

    Once saved, an agent can execute the workflow using ghost_run with specific parameters.

    Example:

    User: "Send an email to sarah@company.com about the Q4 report"
Agent: ghost_run recipe:"gmail-send" params:{recipient:"sarah@company.com", subject:"Q4 report", body:"..."}

  4. Understand the Ghost OS architecture

    main

    Ghost OS is an MCP (Model Context Protocol) server written in Swift that enables AI agents (such as Claude Code, Cursor, or any MCP client) to interact with macOS. It uses the stdio transport layer to communicate via the MCP protocol.

    The architecture consists of several core components:

    • Perception: Uses the Accessibility (AX) tree to see what is on the screen.
    • Vision: Provides visual grounding using a local ShowUI-2B model.
    • Actions: Executes user-like interactions including click, type, scroll, and keys.
    • Recipes: Manages self-learning workflows.
    • AXorcist: The underlying macOS accessibility engine that powers perception and interaction.
    AI Agent (Claude Code, Cursor, any MCP client)
    │
    │ MCP Protocol (stdio)
    │
Ghost OS MCP Server (Swift)
    │
    ├── Perception ──── see what's on screen (AX tree)
    ├── Vision ──────── visual grounding (ShowUI-2B, local)
    ├── Actions ─────── click, type, scroll, keys
    ├── Recipes ─────── self-learning workflows
    └── AXorcist ────── macOS accessibility engine

  5. Install Ghost OS

    main

    Ghost OS can be installed via Homebrew or manually. After installation, you must run ghost setup to configure permissions, MCP configuration, recipe installation, and the vision model setup.

    Homebrew Installation

    brew install ghostwright/ghost-os/ghost-os
ghost setup

    Manual Installation (for macOS developer betas)

    If Homebrew fails due to Xcode version issues on macOS betas, use the following sequence:

    curl -sL https://github.com/ghostwright/ghost-os/releases/latest/download/ghost-os-2.2.1-macos-arm64.tar.gz | tar xz
sudo cp ghost /opt/homebrew/bin/
sudo cp ghost-vision /opt/homebrew/bin/
sudo mkdir -p /opt/homebrew/share/ghost-os
sudo cp GHOST-MCP.md /opt/homebrew/share/ghost-os/
sudo cp -r recipes /opt/homebrew/share/ghost-os/
sudo cp -r vision-sidecar /opt/homebrew/share/ghost-os/
ghost setup
    brew install ghostwright/ghost-os/ghost-os
ghost setup

  6. Best practices for using the `type` tool in recipes

    main
    When creating recipes that involve typing into specific fields, using more descriptive field names improves reliability. For example, in Gmail-related recipes, using into:"To recipients" is more reliable than using into:"To".

  7. Use Self-Learning Mode to record workflows

    main

    Ghost OS can learn workflows by observing a user perform a task. To use this mode:

    1. Call ghost_learn_start with a task_description.
    2. Instruct the user to perform the task manually.
    3. Once the user is finished, call ghost_learn_stop to retrieve the array of recorded actions.
    4. Analyze the actions to identify parameters (like emails or URLs) and synthesize a Recipe JSON.

    Requirements & Constraints:

    • Permissions: Requires 'Input Monitoring' permission.
    • Privacy: Password fields are automatically redacted; recordings are ephemeral (in-memory only) and never written to disk.
    • Concurrency: Do not call ghost_run while ghost_learn is active, as synthetic events will be re-recorded.
    • Scope: Only actions occurring between ghost_learn_start and ghost_learn_stop are recorded.

  8. Orient yourself using ghost_context

    main

    To avoid interacting with the wrong element, always call ghost_context with the target app name before acting. This tool provides critical state information, including:

    • The active app and window
    • The current URL (for browsers)
    • The currently focused element
    • Visible interactive elements

  9. Use Ghost OS Recipes for multi-step tasks

    main
    Before performing any multi-step task manually, check if a pre-tested recipe exists using ghost_recipes. If a recipe is found, execute it using ghost_run with the appropriate recipe name and parameters. Recipes are more reliable and faster than manual tool sequences.

  10. Reliable web app interaction using dom_id

    main

    In web applications (Chrome/Electron), elements are often exposed as generic AXGroup, making ghost_find by name unreliable. The most reliable method is to find the dom_id first and then click using that ID.

    Pattern:

    1. Use ghost_find to retrieve the dom_id of the element.
    2. Use ghost_click with the dom_id parameter.
    ghost_find query:"Send" role:AXButton app:"Chrome"  -> get dom_id from result
ghost_click dom_id:":oq" app:"Chrome"               -> click by dom_id

  11. Recipe JSON Schema (v2)

    main

    All recipes saved via ghost_recipe_save must follow the version 2 schema. Missing or misnamed fields will cause a decode error.

    Top-Level Fields

    • schema_version (Required): Must be 2.
    • name (Required): String used as the filename and identifier.
    • description (Required): String shown in ghost_recipes listings.
    • steps (Required): An array containing at least one step.
    • app (Optional): The default application for all steps.
    • params (Optional): Definitions for runtime parameters.
    • preconditions (Optional): Conditions to check before execution (e.g., app_running, url_contains).
    • on_failure (Optional): Either "stop" (default) or "skip".
    {
  "schema_version": 2,
  "name": "my-recipe-name",
  "description": "What this recipe does",
  "app": "Google Chrome",
  "params": {
    "recipient": {
      "type": "string",
      "description": "Email address of recipient",
      "required": true
    }
  },
  "preconditions": {
    "app_running": "Google Chrome",
    "url_contains": "example.com"
  },
  "steps": [
    {
      "id": 1,
      "action": "click",
      "target": {
        "criteria": [{"attribute": "AXDOMIdentifier", "value": ":oq"}],
        "computedNameContains": "Compose"
      }
    }
  ],
  "on_failure": "stop"
}

  12. Run the Ghost OS Vision Sidecar server

    main

    The Vision Sidecar is an HTTP server used for VLM (Vision Language Model) grounding and element detection. It typically runs on localhost:9876. You can start it using the following commands:

    # Start with default settings (port 9876, auto-detect model)
python3 server.py

# Start on a custom port
python3 server.py --port 9877

# Specify an explicit model path
python3 server.py --model-path /path/to/model

# Pre-load the VLM model at startup (instead of lazy-loading on first request)
python3 server.py --preload

# Run a health check to verify model loading, then exit
python3 server.py --health-check
    python3 server.py