OpenAdapt

The openadapt-tray package is a cross-platform system tray application designed to provide a graphical user interface (GUI) for the OpenAdapt ecosystem. It acts as an orchestration layer that allows users to perform the following tasks without using the command line:

• Control recording: Start and stop workflow recordings.
• Monitor training: Track the progress of training sessions.
• View captures: Access and review recorded data.
• Access settings: Manage application configurations.

OpenAdapt is composed of several specialized repositories. To use the core product, you should focus on the following components:

• openadapt (this repository): Provides the installer, unified CLI, and release compatibility.
• openadapt-flow: The canonical compiler, governed runtime, CLI implementation, and conformance tests. This is where new product and engine development occurs.
• openadapt-desktop: A native application used to record, inspect, qualify, execute, and review workflows.
• Documentation: Comprehensive guides for installation, workflow authoring, qualification, operation, and deployment can be found at https://docs.openadapt.ai.

Note on Legacy Versions: The pre-1.0 monolith is available in the legacy/ directory. For migration, refer to docs/LEGACY_FREEZE.md.

OpenAdapt is designed to be privacy-respecting. It follows a strict policy of collecting only ethical data (error types, function timing, feature names) and explicitly avoiding sensitive information.

What is NEVER collected:

• PII: Names, emails, IP addresses.
• Sensitive Content: Screenshots, user-typed text, file contents, credentials (API keys, passwords), and network request bodies.
• System Metadata: Full file paths (usernames are scrubbed).

Automatic Scrubbing: The telemetry client uses a PII_DENYLIST to automatically scrub keys such as password, secret, token, api_key, authorization, cookie, session, email, phone, address, ssn, and credit_card from all events. File paths are also sanitized to remove usernames (e.g., /Users/john/code/file.py becomes /Users/<user>/code/file.py).

OpenAdapt organizes data and actions through a progressive abstraction ladder. This defines how raw user interactions are transformed into high-level semantic goals.

OpenAdapt uses a unified telemetry layer via the openadapt-telemetry package. This package acts as a shared dependency for all OpenAdapt packages (like openadapt-capture, openadapt-ml, etc.), providing a consistent interface for error tracking, performance monitoring, and feature usage.

Key Components:

• openadapt-telemetry: The central module providing the public API, configuration management, and a Sentry-compatible client.
• Privacy Filter: Ensures PII (Personally Identifiable Information) is scrubbed before transmission.
• Backend: Uses GlitchTip (compatible with the Sentry Python SDK) for error tracking and monitoring.

The application uses a state machine to manage transitions between different operational modes. The TrayState enum defines the possible states:

• IDLE (Default)
• RECORDING_STARTING / RECORDING / RECORDING_STOPPING
• TRAINING / TRAINING_PAUSED
• ERROR

Developers can use StateManager to transition states and register listeners that react to state changes (e.g., updating the tray icon or showing notifications).

from openadapt_tray.state import StateManager, TrayState

manager = StateManager()

def on_change(state):
    print(f"New state: {state.state}")

manager.add_listener(on_change)
manager.transition(TrayState.RECORDING)

OpenAdapt uses a 'fail-closed' and 'governed' model for high-stakes automation:

• Fail-closed execution: Before an action, the runtime checks authorization, identity, and application state. After an action, it waits for settled state and evaluates the declared effect. If the contract cannot be established, it halts and provides evidence.
• Deterministic healthy runs: The runtime uses structural, accessibility, visual, OCR, and spatial signals. Generative models are not part of the 'healthy' path but may be used for Governed Repair.
• Governed repair: Repairs are versioned changes rather than improvisations. Candidate repairs must be reviewed, tested against the qualification contract, and can be promoted or rolled back.

Demonstrations are stored in a structured directory format containing metadata, action data in Parquet format, and screenshots in an observations/ subdirectory.

demonstrations/
  my-task/
    metadata.json       # Session metadata
    actions.parquet     # Action data (observation-action pairs)
    observations/       # Screenshot images (observations)
      0001.png
      0002.png
      ...

Unlike standard automation that only confirms a 'click' was successful, OpenAdapt separates action delivery from outcome verification.

Workflows can bind consequential writes to an independent interface, a separate read-only session, or persisted-state reacquisition. A run is only reported as VERIFIED once the declared business effect is confirmed through these independent channels, ensuring the intended transaction actually committed.

OpenAdapt utilizes several feedback loops to improve agent performance and system reliability. These loops connect the Demonstrate, Learn, and Execute phases.

The OpenAdapt Tray uses a local-first architecture for logging and action storage to ensure privacy and performance. The architecture consists of:

• Local Logging: Logs are stored in platform-specific paths and follow strict rotation and retention policies to manage disk space.
• Action History: User interactions are stored in a SQLite-based database, which is synchronized with the capture database.
• Telemetry: Includes integration with a shared telemetry module for crash reporting and opt-in analytics (refer to the telemetry design document for implementation details).
• Privacy: Designed with a local-first approach that avoids PII (Personally Identifiable Information) collection and provides clear opt-in/opt-out controls.
• Storage: Uses an organized directory structure that follows standard OS conventions.