OpenClicky Documentation

repository·main·Indexed 19 days ago

https://github.com/jasonkneen/openclicky

A native macOS menu-bar companion hosting 'Clicky', an AI agent for voice help, screen-aware guidance, and local automation. Features include the cua-driver skill for accessibility-driven macOS app control, a local HTTP External Control Bridge API, and integrations with Google Workspace via gogcli and the Airtable REST API. Supports local-first configuration and privacy-focused secrets management.

Tokens
279K
Snippets
545
Records
1K
Agent score
66%

What's inside OpenClicky

  1. Overview of the Spotify toolset

    main

    The Spotify skill provides 7 specialized tools to control a user's Spotify account, including playback, device management, queueing, searching, playlist management, album retrieval, and library operations.

    Note on Subscription Levels:

    • Playback-mutating actions (play, pause, skip, etc.) require a Spotify Premium account.
    • Read-only operations (search, library, playlists) work on Spotify Free accounts.
    Prerequisites tools:
    - `spotify_playback` 
    - `spotify_devices` 
    - `spotify_queue` 
    - `spotify_search` 
    - `spotify_playlists` 
    - `spotify_albums` 
    - `spotify_library`
  2. Overview of the cua-driver Computer Use skill

    main

    The cua-driver skill is an OpenClicky bundled capability that enables Codex agents to drive native macOS applications using a local computer-use MCP server.

    Key Capabilities:

    • Accessibility-driven automation: Agents snapshot an app's accessibility tree and perform actions via element_index.
    • Backgrounded-first execution: By default, the skill operates without stealing focus, warping the cursor, or following the user across macOS Spaces.
    • Verification: Uses a 'snapshot-before-AND-after' invariant to ensure actions are successfully executed.
    • Specialized workflows: Includes recipes for backgrounded clicks (using yabai and SLEventPostToPid) and handling web-app quirks in Chromium/WebKit/Electron/Tauri environments.

    Important Note for Users: Do not attempt to install CuaDriver.app or run the standalone cua-driver CLI manually. OpenClicky manages its own helper binary (ClickyComputerUseRuntime) and inherits necessary Accessibility and Screen Recording permissions automatically.

  3. Use gog CLI for Google Workspace automation

    main

    The gog CLI is the primary local route for interacting with Google Workspace (Gmail, Calendar, Drive, Docs, Sheets, Chat, Contacts, etc.) within OpenClicky. It is preferred over browser automation for standard requests.

    Important Constraints:

    • Scope: Use gog for Google Workspace. Do not use it for Google Cloud/GCP, Google Ads, generic Google Search, or public web research.
    • Safety: By default, gog should be treated as read-only. Writes (sends, deletes, moves, etc.) require explicit user approval.
    • Local Paths:
      • CLI: gog (Homebrew: /opt/homebrew/bin/gog)
      • Config: ~/Library/Application Support/gogcli/config.json
      • Credentials: ~/Library/Application Support/gogcli/credentials.json
      • Keyring: ~/Library/Application Support/gogcli/keyring/
    # Example of checking gog version
    ```bash
    gog --version
  4. Use the Maps Skill for location intelligence

    main

    The Maps Skill provides location intelligence using OpenStreetMap, Nominatim, Overpass API, OSRM, and TimeAPI.io. It allows for geocoding, reverse geocoding, finding points of interest (POIs), calculating travel distances/times, and obtaining turn-by-turn directions. It requires Python 3.8+ or curl and has no external API key requirements.

    Key Capabilities:

    • Geocoding: Convert place names to coordinates (search).
    • Reverse Geocoding: Convert coordinates to addresses (reverse).
    • POI Discovery: Find nearby places by category (nearby).
    • Routing: Calculate distance/time (distance) or get turn-by-turn steps (directions).
    • Timezones: Get local time and UTC offsets (timezone).
    • Area Analysis: Get bounding boxes for places (area) or search within a specific rectangle (bbox).
  5. Understand the OpenClicky Roadmap and Priority Levels

    main

    The OpenClicky architecture roadmap categorizes planned improvements and fixes by priority (P0 to P2) and estimated effort (Quick, Short, Medium, Large).

    Priority Levels:

    • P0: Critical fixes and core architectural requirements (e.g., honoring realtime tool-call classification, fixing background computer use selectors, and extracting the VoiceRouter).
    • P1: Feature enhancements and technical debt reduction (e.g., bumping Anthropic defaults, implementing a SkillCorpus abstraction, and introducing AgentExecutor protocols).
    • P2: Long-term stability and advanced features (e.g., resolving runtime ambiguity, fixing process lifecycles, and implementing model-authoritative routing for push-to-talk).

    Effort Estimates:

    • Quick: < ½ day
    • Short: ≤ 1 day
    • Medium: 2-4 days
    • Large: 1-2 weeks
  6. Understand Hallmark's design token export formats

    main

    Hallmark emits design system tokens in four canonical formats to ensure portability across different project types.

    1. tokens.css: The primary source of truth. It contains plain CSS custom properties at :root. This file is always generated and should be imported by your page CSS.
    2. Tailwind v4 @theme: For Tailwind projects. These are provided inline in design.md for easy copying. It maps Hallmark tokens to Tailwind-compatible variables (e.g., mapping --space-* to --spacing-*).
    3. DTCG tokens.json: For projects using token pipelines like Style Dictionary, Token Studio, or Cobalt. This follows the W3C Design Tokens Community Group format.
    4. shadcn/ui CSS variables: For projects using the shadcn/ui component library. These are also provided inline in design.md.

    Note: tokens.css is a side effect of every build and is always written to disk. The other three formats are embedded in design.md for manual extraction.

  7. Query Polymarket prediction market data

    main

    You can query prediction market data from Polymarket, including markets, prices, orderbooks, and history, using their public REST APIs. All endpoints are read-only and require zero authentication.

    When to use this integration:

    • When a user asks about prediction markets, betting odds, or event probabilities.
    • When a user asks for market prices, orderbook data, or price history.
    • When a user wants to monitor or track prediction market movements.

    Key Concepts:

    • Events vs. Markets: An Event contains one or more Markets (1:many relationship).
    • Prices as Probabilities: Markets are binary outcomes with Yes/No prices between 0.00 and 1.00. A price of 0.65 represents a 65% probability.
    • Volume: Measured in USDC (US dollars).

    API Endpoints:

    • Gamma API (gamma-api.polymarket.com): Used for discovery, search, and browsing.
    • CLOB API (clob.polymarket.com): Used for real-time prices, orderbooks, and history.
    • Data API (data-api.polymarket.com): Used for trades and open interest.

    Note: For full endpoint references and curl examples, see references/api-endpoints.md in the repository.

  8. Overview of the Spotify Skill tools

    main

    The Spotify skill provides control over playback, devices, queues, search, playlists, albums, and the user's library through 7 specialized tools.

    Note on Spotify Tiers:

    • Playback-mutating actions (play, pause, skip, etc.) require Spotify Premium.
    • Search, library, and playlist operations work on the Free tier.
    ### The 7 tools
    
    - `spotify_playback` — play, pause, next, previous, seek, set_repeat, set_shuffle, set_volume, get_state, get_currently_playing, recently_played
    - `spotify_devices` — list, transfer
    - `spotify_queue` — get, add
    - `spotify_search` — search the catalog
    - `spotify_playlists` — list, get, create, add_items, remove_items, update_details
    - `spotify_albums` — get, tracks
    - `spotify_library` — list/save/remove with `kind: "tracks"|"albums"`
  9. Follow OpenClicky specialist agent guardrails

    main

    When developing or managing specialist agents, adhere to these safety and quality constraints:

    • Archiving: Never overwrite a user-authored agent or skill without archiving the prior version first.
    • Scope: Avoid creating broad or vague agents. Specialists must have a narrow job and named skills.
    • Skill Verification: Do not claim a specialist can use a skill unless it is explicitly enabled in skills.json or exists in the agent's custom skills/ folder.
    • Error Reporting: If a tool, auth, or macOS permission is missing, stop and report the exact missing setup step. Do not loop or silently switch to browser automation.
    • Write Safety: Treat all external actions (sends, publishes, deploys, deletes, moves, merges, cloud writes, etc.) as external writes unless the specific skill narrows that scope.
  10. Follow content and anti-slop design discipline

    main

    Maintain high design fidelity by avoiding 'AI design sludge' and filler content.

    Content Discipline:

    • Do not add fake metrics, decorative stats, generic feature grids, or placeholder testimonials.
    • Every element must earn its place. If a new section is needed to improve the artifact, ask the user first.
    • Mark non-final copy as draft or placeholder.

    Anti-Slop Rules (Avoid these):

    • Aggressive gradient backgrounds and default glassmorphism.
    • Generic SaaS cards with ubiquitous icons.
    • Left-border accent callout cards.
    • Fake dashboards with arbitrary numbers.
    • Stock-photo hero sections.
    • Oversized rounded rectangles used as a substitute for hierarchy.
    • Vague labels like "Insights," "Growth," or "Scale" without supporting content.
  11. Compare Image mode vs URL mode in `hallmark study`

    main

    When using hallmark study, the source of the input changes what information can be extracted. URL mode provides higher technical fidelity for most steps but has a specific blind spot regarding visual rhythm.

    StepImage modeURL mode
    1 Surfacecolor bands and footprint, estimated by eyeexact OKLCH / hex / rgb values from CSS
    2 Typeroles only (e.g., "italic editorial serif")roles plus exact font names from @font-face or Google Fonts
    3 Structureinferred from visible regionsinferred from real DOM (<nav>, <section>, etc.)
    4 Motionusually "not visible — assuming default reveals"observable via <script> tags (e.g., framer-motion, gsap) and CSS @keyframes
    5 Rhythmobservable directly from visual gestaltnot observable (marked as unknown (URL mode) in schema)

    Note: If extracting visual rhythm is a priority, provide a screenshot instead of a URL.