Mac Mini Agent

repository·main·Indexed 42 days ago

https://github.com/disler/mac-mini-agent

A toolkit for macOS automation designed for AI agents, providing a bridge between terminal-based agents and full computer use. It includes Steer (v0.2.0) for GUI control via screenshots and OCR, Drive (v0.1.0) for tmux session management, and a remote execution system consisting of the Listen server and Direct CLI client (v0.1.0).

Tokens
15.8K
Snippets
80
Records
93
Agent score
54%

What's inside Mac Mini Agent

  1. Overview of the Listen & Direct Remote Agent Job System

    main

    The Listen & Direct system enables remote agent execution across different machines over HTTP. It consists of two primary components:

    1. apps/listen (The Server/Worker): A lightweight HTTP server and job manager that runs on the target machine (e.g., a Mac Mini). It accepts job requests, spawns autonomous worker processes that run agents (like Claude Code), and manages job state using YAML files on disk.
    2. apps/direct (The CLI Client): A command-line interface used to interact with a listen server. It allows you to start, check, list, and stop jobs remotely.

    Workflow:

    • A client uses direct start to send a prompt to the listen server via POST /job.
    • The server immediately returns a job_id and spawns a background worker.
    • The worker executes the agent, which periodically updates a YAML file with progress and a final summary.
    • The client polls GET /job/{job_id} to retrieve the job's current state and results.
  2. How Cross-App Pipelines work

    main

    You can combine Steer (GUI) and Drive (Terminal) to build automated pipelines that span multiple applications without human intervention.

    Example Workflow:

    1. Read a GitHub issue using a browser via steer ocr.
    2. Execute code changes using Claude Code in a terminal via drive run.
    3. Log the results into Notion using steer to interact with the Notion GUI.
  3. How Agent-on-Agent Orchestration works

    main
    A primary AI agent can control other AI agents through tmux. Using the Drive CLI, a master agent can spin up tmux sessions, launch secondary agents within those sessions, send them prompts, monitor their output, and manage their lifecycle (closing them when tasks are complete). This allows for complex, multi-agent workflows where one agent acts as a manager for several specialized workers.
  4. Implement the Agent Loop pattern with `steer`

    main

    The recommended workflow for an AI agent using steer follows an Observe-Act-Verify loop:

    1. Observe: Use steer see --json to get a screenshot and the current accessibility element map.
    2. Act: Use steer click, steer type, or steer hotkey to perform an action based on the observation.
    3. Verify: Use steer see --json again to confirm the action resulted in the expected UI change.
    steer see --json          # 1. observe — screenshot + element map
    steer click --on B3       # 2. act — click, type, hotkey
    steer see --json          # 3. verify — see the result
  5. Install and build Steer

    main

    To set up the steer CLI, run the provided setup.sh script, then build the project using Swift. The resulting binary will be located at .build/release/steer.

    Setup Steps

    1. Run the setup script:
      chmod +x setup.sh
      ./setup.sh
    2. Build the release binary:
      swift build -c release

    Verification

    • Check the binary exists: .build/release/steer
    • Check help output: .build/release/steer --help (should show 6 subcommands: see, click, type, hotkey, scroll, apps).
    chmod +x setup.sh
    ./setup.sh
    swift build -c release
  6. Use Drive for Terminal and tmux Control

    main

    Drive is a Python-based CLI (v0.1.0) that provides programmatic control over tmux sessions. It allows agents to create terminals, execute commands, and orchestrate parallel workloads.

    Key Commands:

    • session: Create, list, and manage tmux sessions.
    • run: Execute a command in a tmux pane and wait for completion.
    • send: Send keystrokes to a tmux pane.
    • logs: Capture pane output using capture-pane.
    • poll: Wait for a sentinel marker indicating command completion.
    • fanout: Execute commands across multiple panes in parallel.

    The Sentinel Pattern: To make asynchronous terminal orchestration deterministic, Drive uses a sentinel marker: __DONE_<token>:<exit_code>. This allows the agent to reliably detect when a command has finished instead of relying on guesswork.

    # Create a new tmux session
    drive session create --name agent-1
    
    # Run a command and wait for it to finish
    drive run --session agent-1 "npm test"
    
    # Capture the output of a session
    drive logs --session agent-1
    
    # Run a command across multiple sessions in parallel
    drive fanout --sessions a1,a2,a3 "build"
  7. Set up the Primary Device (Engineer Devbox)

    main

    Your primary Mac (laptop) is used to send prompts and monitor jobs. It requires minimal dependencies (uv, just) and the CLI client tools.

    Automated Installation

    Run the installer on your primary Mac, providing the IP address or hostname of the Mac Mini sandbox:

    /install-engineer-devbox <mac-mini-ip-or-hostname>

    This command installs dependencies, configures the sandbox URL, and runs a 6-check verification suite (network connectivity, listen server, SSH access, and end-to-end job test).

    Managing Jobs via CLI

    After setup, use the following just commands to interact with the sandbox:

    • just send "<prompt>": Send a new job to the agent.
    • just jobs: List all active/past jobs.
    • just job <id>: Check the status of a specific job.
    • just stop <id>: Terminate a running job.

    Debugging via SSH

    If you need to debug the sandbox directly, you can SSH into the Mac Mini:

    ssh <user>@<mac-mini-ip>
    # or via Bonjour
    ssh <user>@<hostname>.local
  8. Use Listen and Direct for Remote Agent Execution

    main

    The Listen and Direct pair forms a job management system for remote agent execution.

    Listen (Job Server)

    A FastAPI-based Python server (v0.1.0) that accepts job requests over HTTP, spawns agents (like Claude Code) as worker processes, and tracks progress via YAML job files.

    HTTP Endpoints:

    • POST /job: Submit a prompt; returns a job ID.
    • GET /job/{id}: Check job status, updates, and summary.
    • GET /jobs: List all jobs.
    • DELETE /job/{id}: Stop a running job.

    Direct (CLI Client)

    A Python CLI (v0.1.0) using httpx and Click to interact with the Listen server.

    Direct Commands:

    • start: Submit a new job prompt.
    • get: Get current state of a job.
    • list: List all jobs.
    • latest: Show full details of the latest N jobs.
    • stop: Stop a running job.

    Workflow Example:

    # Start the Listen server (typically on port 7600)
    just listen
    
    # Submit a job using Direct
    just send "Open Safari..."
    
    # Monitor jobs
    just jobs
    just job <id>
    just latest
    # Start server
    just listen
    
    # Submit a job
    just send "Open Safari..."
    
    # List all jobs
    just jobs
    
    # Check a specific job
    just job <id>
    
    # Get full details of the most recent job
    just latest
  9. Configure macOS Permissions for steer

    main

    steer requires specific macOS privacy permissions to function:

    1. Accessibility: Required for accessing the accessibility tree and sending input events.
    2. Screen Recording: Required for taking screenshots.

    steer will auto-prompt for Accessibility on its first run. You must manually grant both permissions in System Settings > Privacy & Security.

  10. Set up the Mac Mini Device (Agent Sandbox)

    main

    The Mac Mini acts as the agent's workstation, running the Listen server and providing full GUI and terminal access.

    Automated Installation

    Run the following command directly on the Mac Mini to install dependencies (Swift, tmux, uv, just, yq, Claude Code), build steer, set up Python environments, and run a 10-check verification suite:

    /install-agent-sandbox

    Manual Permissions Required

    You must manually grant the following permissions in System Settings for the agent to function correctly:

    PermissionPurposeHow to configure
    AccessibilityAllows steer to click, type, and read UI elementsSystem Settings > Privacy & Security > Accessibility > add Terminal
    Screen RecordingAllows steer to capture screenshots (steer see)System Settings > Privacy & Security > Screen Recording > add Terminal
    Full Disk AccessRequired for systemsetup and broad file accessSystem Settings > Privacy & Security > Full Disk Access > add Terminal
    Remote Login (SSH)Enables remote management of the sandboxSystem Settings > General > Sharing > toggle Remote Login on

    Starting the Server

    Once installation is complete, start the agent server using just:

    just listen    # Starts Listen on port 7600