DOCSEARCHbyAlgolia
Submit Docs

MacosUseSDK

repository·main·Indexed 41 days ago

https://github.com/mediar-ai/macosusesdk

A Swift library and set of command-line tools for traversing the macOS accessibility tree and simulating user input. It includes utilities for opening applications (AppOpenerTool), inspecting UI structures (TraversalTool), visual debugging (HighlightTraversalTool), and simulating mouse and keyboard events (InputControllerTool and VisualInputTool). The SDK supports UI automation, testing, and accessibility structure analysis with features like traversal diffing and visual feedback.

Tokens
5.5K
Snippets
21
Records
23
Agent score
52%

What's inside MacosUseSDK

  1. Build the MacosUseSDK tools

    main

    To compile the command-line tools, navigate to the root directory (MacosUseSDK) and use the Swift build system. Executables will be located in .build/debug/ (or .build/release/ if using the -c release flag). You can run them via the path or using swift run <ToolName>.

    swift build

  2. Integrate MacosUseSDK into a Swift project

    main

    You can use MacosUseSDK as a Swift Package dependency. Add it to your Package.swift and import it into your targets. When using asynchronous UI functions like clickMouseAndVisualize or highlightVisibleElements, ensure you are on the main thread and keep the RunLoop active.

    // Package.swift
dependencies: [
    .package(url: "/* path or URL to your MacosUseSDK repo */", from: "1.0.0"),
]

// Target configuration
.target(
    name: "YourApp",
    dependencies: ["MacosUseSDK"]
)

// Usage
import MacosUseSDK
import Foundation

Task {
    do {
        let point = CGPoint(x: 100, y: 200)
        try MacosUseSDK.clickMouse(at: point)

        DispatchQueue.main.async {
            do {
                 try MacosUseSDK.clickMouseAndVisualize(at: point, duration: 1.0)
            } catch {
                 print("Visualization error: \(error)")
            }
        }
    } catch {
        print("MacosUseSDK Error: \(error)")
    }
}

  3. Inspect TraversalDiff results

    main

    The ActionResult returned by performAction may contain a traversalDiff. This object describes the changes in the UI hierarchy between the traverseBefore and traverseAfter steps.

    If showDiff was enabled in ActionOptions, the traversalDiff contains lists of:

    • added: Elements that appeared in the UI.
    • removed: Elements that disappeared from the UI.
    • modified: Elements whose attributes (like text, role, etc.) changed.

    For modified elements, the changes array provides specific details on which attribute changed, including oldValue and newValue (or addedText/removedText for text-specific changes).

  4. Handle asynchronous visual feedback in CLI tools

    main

    When using MacosUseSDK within a command-line tool, visual feedback functions (like showVisualFeedback or drawHighlightBoxes) schedule work on the main thread asynchronously via DispatchQueue.main.async.

    Because these functions return immediately after scheduling the work, a CLI tool that calls exit(0) immediately after performAction may terminate before the animations or highlights are actually rendered.

    Requirement: To ensure animations are visible, introduce a deliberate pause (e.g., Task.sleep) at the end of your main logic before exiting.

    // After performAction calls are complete
fputs("info: Pausing to allow async animations to complete...\n", stderr)
try? await Task.sleep(nanoseconds: 1_000_000_000) // 1 second pause
exit(0)

  5. Configure action behavior with ActionOptions

    main

    The ActionOptions struct controls how an action is executed and how the UI state is analyzed before and after the action.

    Key configuration properties:

    • traverseBefore: If true, the SDK performs a UI traversal before the action to capture the initial state.
    • traverseAfter: If true, the SDK performs a UI traversal after the action to capture the resulting state.
    • showDiff: If true, calculates and provides a traversalDiff representing the changes in the UI.
    • onlyVisibleElements: If true, limits traversal to elements currently visible to the user.
    • showAnimation: If true, enables visual feedback (like highlights or captions).
    • animationDuration: The duration (in seconds) for visual animations.
    • pidForTraversal: The Process ID of the application to target for UI traversal.
    • delayAfterAction: A delay (in seconds) to wait after the primary action completes but before the traverseAfter step occurs.

  6. Use VisualInputTool for input with visual feedback

    main

    The VisualInputTool simulates keyboard and mouse input events but adds a visual feedback effect (a pulsing green circle for mouse actions). Use the --duration <seconds> option to specify the effect duration (default: 0.5s). Note that keypress and writetext currently do not implement visualization.

    # Left click at (100, 250) with default 0.5s feedback
swift run VisualInputTool click 100 250

# Right click at (800, 400) with 2 second feedback
swift run VisualInputTool rightclick 800 400 --duration 2.0

  7. Use HighlightTraversalTool for visual debugging

    main

    The HighlightTraversalTool traverses an application's accessibility tree (by PID) and draws temporary red boxes around all visible UI elements while also outputting the JSON traversal data to stdout. This is useful for debugging accessibility structures. Use the --duration <seconds> option to control how long the highlights stay visible (default is 3.0s).

    # Combine with AppOpenerTool to open Messages and highlight it
swift run HighlightTraversalTool $(swift run AppOpenerTool Messages) --duration 5

  8. Use TraversalTool to inspect the accessibility tree

    main

    The TraversalTool traverses the accessibility tree of a running application (identified by PID) and outputs a JSON representation of the UI elements to stdout. Use the --visible-only flag to filter for elements that have a geometric position and size.

    # Get only visible elements for Messages app
swift run TraversalTool --visible-only $(swift run AppOpenerTool Messages)

  9. Use AppOpenerTool to open applications

    main

    The AppOpenerTool opens or activates a macOS application using its name, bundle ID, or full path. On success, it outputs the application's PID to stdout. Informational logs are sent to stderr.

    # Open by name
swift run AppOpenerTool Calculator

# Open by bundle ID
swift run AppOpenerTool com.apple.Terminal

# Open by path
swift run AppOpenerTool /System/Applications/Utilities/Terminal.app

  10. Use InputControllerTool to simulate input

    main

    The InputControllerTool simulates keyboard and mouse events without visual feedback.

    Available Actions:

    • keypress <key>: e.g., enter, cmd+c, shift+tab
    • click <x> <y>: Left click at coordinates
    • doubleclick <x> <y>: Double click at coordinates
    • rightclick <x> <y>: Right click at coordinates
    • mousemove <x> <y>: Move cursor to coordinates
    • writetext <string>: Type text
    # Press the Enter key
swift run InputControllerTool keypress enter

# Simulate Cmd+C (Copy)
swift run InputControllerTool keypress cmd+c

# Left click at screen coordinates (100, 250)
swift run InputControllerTool click 100 250

# Type the text "Hello World!"
swift run InputControllerTool writetext "Hello World!"

  11. Call traverseAccessibilityTree from MacosUseSDK

    main

    The MacosUseSDK library provides the core logic for tree traversal via the traverseAccessibilityTree function. This function is designed to be called programmatically to retrieve accessibility data for a specific process.

    Parameters:

    • pid: The Int32 process ID of the target application.
    • onlyVisibleElements: A Bool indicating whether to filter for only visible elements.

    Returns:

    • A response object (encodable to JSON) containing the accessibility tree data.

    Errors:

    • Throws MacosUseSDKError if the traversal fails.
    // Example usage within a Swift application
do {
    let responseData = try MacosUseSDK.traverseAccessibilityTree(pid: appPID, onlyVisibleElements: onlyVisible)
    // responseData can then be encoded to JSON
} catch let error as MacosUseSDKError {
    print("Error from MacosUseSDK: \(error.localizedDescription)")
} catch {
    print("An unexpected error occurred: \(error.localizedDescription)")
}

  12. Perform user actions with performAction()

    main

    The performAction(action:optionsInput:) function is the primary entry point for executing user-like interactions (such as opening applications or typing text) using the MacosUseSDK. It takes a PrimaryAction and an ActionOptions configuration object.

    When performing actions that involve UI changes (like typing), it is highly recommended to use the pidForTraversal option to ensure the SDK targets the correct application process for state comparison and visual feedback.

    let openOptions = ActionOptions(
    traverseBefore: true,
    traverseAfter: true,
    showDiff: false,
    onlyVisibleElements: true,
    showAnimation: false,
    delayAfterAction: 0.0
)

let openResult = await performAction(action: PrimaryAction.open(identifier: "TextEdit"), optionsInput: openOptions)

// For subsequent actions like typing, use the PID from the previous result
let typeOptions = ActionOptions(
    traverseBefore: true,
    traverseAfter: true,
    showDiff: true,
    onlyVisibleElements: true,
    showAnimation: true,
    animationDuration: 0.8,
    pidForTraversal: openResult.openResult?.pid,
    delayAfterAction: 0.0
)

let typeResult = await performAction(action: PrimaryAction.input(action: .type(text: "Hello!")), optionsInput: typeOptions)