DOCSEARCHbyAlgolia
Submit Docs

Peekaboo Documentation

repository·main·Indexed 26 days ago

https://github.com/openclaw/peekaboo

A macOS automation tool and MCP server combining high-fidelity screen capture, AI analysis, and GUI automation. It provides a modular set of agent tools for vision, UI interaction, window management, and shell execution, allowing developers to control the macOS interface via CLI or natural language.

Tokens
164.3K
Snippets
418
Records
988
Agent score
89%

What's inside Peekaboo

  1. Overview of supported AI providers in Peekaboo

    main

    Peekaboo supports a variety of AI providers for automation tasks, ranging from cloud-based APIs to local models. Providers include:

    • OpenAI: Supports function/tool calling and vision.
    • Anthropic: Supports Claude models (Fable 5, Sonnet 5, Opus 4.8) with vision and tool calling.
    • Google: Configured via GEMINI_API_KEY; supports Gemini 3.1 Pro Preview and Gemini 3 Flash.
    • MiniMax: Configured via MINIMAX_API_KEY; supports MiniMax M3 and M2.7.
    • MiniMax China: Configured via MINIMAX_CN_API_KEY or MINIMAX_API_KEY; routes to api.minimaxi.com.
    • Kimi: Configured via MOONSHOT_API_KEY or KIMI_API_KEY; supports Kimi K2.6 and K2.7 Code.
    • Grok: Supports Grok 4 implementation.
    • Ollama: Supports local models with incremental NDJSON streaming.
    • LM Studio: Provides an OpenAI-compatible local server for offline use.

  2. Overview of Peekaboo CLI commands

    main

    Peekaboo provides a comprehensive CLI for Mac automation, categorized into core automation primitives, system/configuration management, and MCP (Model Context Protocol) helpers.

    Core Automation

    • Autonomous Agent: agent runs the autonomous agent loop.
    • App & Window Management: app (launch/quit/focus), open (files/URLs), window (move/resize/focus), menu, and menubar (menus/status items).
    • Input Primitives: click, move, scroll, swipe, drag, press, type, set-value, perform-action, hotkey, and sleep.
    • Vision & Capture: see, image, and capture for screenshots, UI maps, and capture sessions.

    System & Configuration

    • Management: config, permissions, bridge, daemon, tools, clean, run, learn, list, and screen.
    • Shell Integration: completions to install zsh, bash, or fish completions.
    • Clipboard: clipboard for clipboard operations.
    • MCP: mcp for Model Context Protocol helpers.

  3. Overview of Peekaboo Service Domains

    main

    Peekaboo services are organized into five functional domains:

    • System Services: OS-level operations including ApplicationService (launching/listing apps), ProcessService (monitoring), and FileService (file operations).
    • UI Services: User interface automation including UIAutomationService (click, type, scroll), WindowManagementService (positioning/focus), MenuService (menu bar), DialogService (alerts), and DockService.
    • Capture Services: Visual analysis via ScreenCaptureService (screenshots with AI element detection).
    • Agent Services: AI-powered automation via PeekabooAgentService and modular Tools.
    • Support Services: Infrastructure including LoggingService, SessionManager, and the PeekabooServices container.

  5. Understand the Ghost Animation System for Menu Bar Icons

    main

    Peekaboo uses a SwiftUI-based 'Ghost Animation System' to provide visual feedback in the macOS menu bar. The system renders an animated ghost icon that indicates the current state of the Peekaboo agent.

    Key Visual Behaviors:

    • Vertical Floating: A sine wave movement with a ±3 pixel amplitude and a 2.5-second cycle.
    • Breathing Effect: Opacity variations between 0.7 and 1.0 over a 2.0-second duration to create an organic appearance.
    • Wavy Bottom Edge: An animated edge effect rendered via SwiftUI Canvas.

    Performance Features:

    • Uses drawingGroup() for optimized rendering.
    • Implements adaptive frame rates (30fps during active animation, 15fps for subtle movement).
    • Utilizes icon caching and quantized animation values to minimize CPU usage and cache misses.

  6. Understand Peekaboo Bridge Host architecture

    main
    Peekaboo Bridge is a socket-based broker designed to handle permission-bound operations like Screen Recording, Accessibility, and AppleScript. It allows a CLI or client process to drive automation by routing requests through a host application (like Peekaboo.app or Claude.app) that already possesses the necessary macOS TCC (Transparency, Consent, and Control) grants. This architecture replaces the previous XPC-based helper approach.

  7. Understand Visual Feedback Animation Types

    main

    Peekaboo uses a "Ghost HUD" design system to provide visual confirmation of automation actions. Key feedback types include:

    • Screenshot Capture: Viewfinder corner brackets snap to the region with a white veil flash.
    • Click Actions: A macOS-style cursor glides to the point. Single clicks emit a ring; double clicks emit two rings; right clicks use a blue secondary accent.
    • Typing: A caption pill at the bottom center streams text verbatim. Special keys are rendered as glyphs (e.g., ⏎, ⇥, ⌫, ⎋).
    • Scrolling: A circular chip with three chevrons flows in the scroll direction.
    • Mouse Movement: A cursor glides with a tapered gradient trail.
    • Gestures: Swipe/Drag actions use a "comet" style with a thicker stroke, using press rings for touch-down and release rings for touch-up.
    • Hotkeys: macOS-style keycaps (e.g., ⌘ command) appear in a HUD chip.
    • Window Operations: Visual outlines indicate actions like Minimize (squash), Maximize/Focus (expand/glow), and Close (contract/fade).
    • Menu Navigation: A breadcrumb chip (e.g., File ▸ New ▸ Project) shows the traversal path.

  8. Understand Swift 6 Concurrency and Data Isolation

    main

    Swift 6 introduces a compiler-enforced concurrency model to prevent data races. The core mechanism is data isolation, which ensures mutually exclusive access to mutable state.

    Declarations exist in one of three isolation domains:

    1. Non-isolated: The default state for functions and variables (e.g., top-level functions or standard classes).
    2. Isolated to an actor value: Properties and methods belonging to an actor instance.
    3. Isolated to a global actor: Declarations statically assigned to a global actor (e.g., @MainActor).

    When moving values between these domains, you are crossing an isolation boundary. Values can only cross if they are safe to share, typically by conforming to the Sendable protocol.

  9. Understand the Peekaboo Modular Architecture

    main

    Peekaboo is transitioning from a monolithic PeekabooCore module to a layered architecture to improve build performance and maintainability. The architecture follows a strict dependency hierarchy where lower layers cannot import higher layers.

    Dependency Layers:

    1. Layer 1: Foundation (PeekabooModels, PeekabooProtocols) - Basic types, enums, errors, and service/tool protocols. No dependencies.
    2. Layer 2: Core Services (PeekabooCapture, PeekabooAutomation, PeekabooSystem, PeekabooVision) - Concrete implementations of services. Depends on Layer 1.
    3. Layer 3: Integration (PeekabooAgent, PeekabooMCP, PeekabooFormatting) - Agent services, MCP integration, and formatters. Depends on Layers 1-2.
    4. Layer 4: Commands (PeekabooCommands) - CLI command implementations (e.g., SeeCommand, ClickCommand). Depends on Layers 1-3.
    5. Layer 5: Application (peekaboo CLI executable) - The top-level entry point (main.swift). Depends on Layer 4.

  10. Understand Peekaboo 3.0 Architecture

    main

    Peekaboo 3.0 is a unified automation stack for macOS. It provides a single set of Swift services that power multiple interfaces:

    • CLI (peekaboo): The primary automation surface for scripts and manual commands.
    • Peekaboo.app: A menu-bar UI providing an inspector, visualizer, and chat/voice interface.
    • Agent Runtime (peekaboo agent): An autonomous interface using LLMs (GPT, Claude, etc.) to perform natural-language tasks.
    • MCP Server (peekaboo mcp): An interface for Model Context Protocol clients like Claude Code or Cursor to use Peekaboo tools.

    All surfaces rely on PeekabooServices to ensure consistent behavior, configuration, and logging.

  11. Understand Peekaboo System Architecture

    main

    Peekaboo is composed of several interconnected modules that handle different aspects of macOS automation:

    • PeekabooAutomation: The core automation layer. It manages configuration, screen capture, application/menu/window services, and snapshot management. It interacts directly with macOS Accessibility and ScreenCaptureKit.
    • PeekabooVisualizer: A visual feedback layer that provides UI feedback (like click animations) via VisualizationClient.
    • PeekabooAgentRuntime: The layer hosting MCP (Model Context Protocol) tools and the agent service. It depends on PeekabooAutomation for services and PeekabooVisualizer for status.
    • PeekabooCore: A thin umbrella module that provides a convenience container (PeekabooServices) for apps and CLI tools to access the underlying services.
    • Tachikoma: The AI model management layer that provides models and streaming capabilities to the runtime modules.

  12. Understand the OpenAI provider architecture in Peekaboo

    main
    The OpenAI provider in Peekaboo uses a protocol-based message and tool abstraction architecture. It has been migrated from the Assistants API to the Chat Completions API. This architecture ensures that streaming, tool calling, and session persistence are handled through a shared model interface, making the provider compatible with the same patterns used for Anthropic, Grok, and Ollama. Tool results and errors are normalized to ensure that agents and CLI renderers remain provider-agnostic.