open-codex-computer-use

This directory serves as a repository for the ongoing reverse engineering analysis of the official closed-source Codex Computer Use.app and SkyComputerUseClient. The goal is to provide traceable inputs for the open-source open-computer-use implementation by documenting confirmed technical details rather than relying on chat histories.

Key areas of research include:

• Architecture & Runtime: Bundle structures, entry points, transport layers, host dependencies, and runtime behaviors.
• IPC & Lifecycle: Internal client-service IPC types, sender authorization, the skyshot model, and turn-ended lifecycle integration.
• Tooling & Interaction: Request/response samples for computer-use tools, background click-free tooling methodologies, and permission onboarding (Accessibility/Screen Recording).
• Software Cursor Implementation: Analysis of the yellow virtual mouse overlay, cursor motion parameter models (including CursorMotionPath.sample(progress), SpringAnimation -> VelocityVerletSimulation timing chains, and geometric scoring), and slider parameter mapping.
• State Rendering: Analysis of state renderer strings, AX fields, tree transforms, and window recovery clues used to converge state outputs for Lark/Electron apps.

The project implements several supply chain security controls to ensure dependency integrity and artifact provenance. The following tools and actions are used:

• Dependency Review: Uses actions/dependency-review-action to block Pull Requests that introduce high-risk dependency changes.
• Vulnerability Scanning: Uses google/osv-scanner-action to scan dependency declarations and lockfiles against known vulnerabilities using OSV.
• SBOM Generation: Uses anchore/sbom-action to generate Software Bill of Materials (SBOM) in SPDX format for release artifacts.
• Build Provenance: Uses actions/attest-build-provenance to generate signed provenance attestations for release artifacts.
• Security Posture Analysis: Uses ossf/scorecard-action to analyze repository-level security signals (e.g., workflow permissions, branch protection).
• Action Pinning: Uses scripts/check-action-pinning.sh to ensure all GitHub Actions are pinned to immutable commit SHAs rather than floating version tags. If a floating tag is detected, the CI will fail.

The project is a local computer-use implementation designed to provide automation capabilities across different operating systems. The core is a Swift-based macOS automation MCP server, with experimental Windows and Linux runtimes implemented in Go that expose the same set of 9 Computer Use tools.

Core Components by Platform

• macOS: Managed via apps/OpenComputerUse, which serves as the main entry point for CLI commands and the macOS app mode. It handles permissions and provides a GUI onboarding window if permissions are missing.
• Windows (Experimental): Managed via apps/OpenComputerUseWindows. It uses a Go CLI/MCP entry point with a PowerShell UI Automation bridge, producing an open-computer-use.exe binary.
• Linux (Experimental): Managed via apps/OpenComputerUseLinux. It uses a Go CLI/MCP entry point with a Python AT-SPI bridge, producing an open-computer-use binary.

Key Packages and Directories

• packages/OpenComputerUseKit: The core library containing the MCP stdio transport, tool registry, app discovery, accessibility/window snapshots, input simulation (keyboard/mouse), and software cursor overlay.
• skills/: Contains installable skills for agent runtimes. The open-computer-use skill is a lightweight entry point, with detailed usage and troubleshooting located in adjacent references/ files.
• apps/OpenComputerUseFixture: A local GUI fixture app used for low-risk, predictable testing of clicks, inputs, scrolling, and dragging.

When using Open Computer Use, adhere to these safety and operational guidelines:

Safety & Privacy

• Session Context: Treat the target desktop as a real user session. Do not inspect password managers or sensitive private content unless explicitly requested.
• Confirmation: Always ask for permission before performing high-stakes actions like sending, deleting, purchasing, approving, or uploading content.

Operational Reliability

• Index Validity: Never guess element_index values. Always run get_app_state immediately before an action to get the most recent mapping.
• Action Selection: Prefer semantic actions and set_value for editable controls. Use coordinate-based click, scroll, and drag only as a fallback when the element tree does not provide a safe target.
• Platform Specifics:
  • macOS: Do not enable OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 unless the user explicitly requests click_method: "global" or diagnostic behavior.
  • Windows/Linux: Ensure the command is running inside a logged-in desktop session to enable GUI automation.

The scripts/codex_dump.py script organizes captured data into three main layers within the <session-name>/ directory. Understanding these layers is key to debugging Codex behavior:

• websocket/*.jsonl: Contains WebSocket frames. Use this to see when the model decides to call a tool and what the parameters are (e.g., response.create, response.output_item.done with item.type=="function_call").
• local-sessions/*.json: Structured summaries exported from ~/.codex/sessions/rollout-*.jsonl. Use this to see which function_call was actually dispatched to a local MCP and what the function_call_output was.
• http/*.json: Standard HTTP request/response pairs. Use this for auxiliary traffic like plugin/config initialization (e.g., wham/apps).

Troubleshooting Priority

When investigating issues, follow this order:

1. Check upstream LLM calls in websocket/.
2. Check local tool/MCP dispatch in local-sessions/.
3. Only if the above are insufficient, check low-level Codex local logs.

To implement a high-fidelity software cursor motion model (mimicking human-like movement), the architecture should be decoupled into four distinct layers rather than being bundled into a single overlay component. This separation allows for independent testing, parameter tuning, and eventual open-sourcing without coupling to the MCP runtime.

1. Motion Parameters

Pure value types that do not depend on AppKit. Essential parameters include:

• startHandle
• endHandle
• arcHeight
• arcFlow
• spring

2. Motion Path Builder

Responsible for geometry only (not time). It takes a start point, end point, and parameters to produce:

• CursorMotionPath
• segments
• control1
• control2
• measurement
• Tangents (切线)

3. Motion Simulator

Handles time progression on top of the path geometry. It manages:

• Bezier progress animation
• Spring simulation
• velocity
• force
• next-interaction timing gate
• completion timing

4. Cursor Renderer / Demo Host

The outermost layer that integrates with AppKit / SwiftUI. It handles:

• Overlay windows
• Debug slider UI
• Target point annotation
• click / mail / debug toggles

To minimize user disruption (mouse hijacking and focus stealing), the project implements a hierarchy of input methods that prioritize non-intrusive channels over global HID (Human Interface Device) events.

Input Hierarchy

1. Accessibility (AX) First: For clicks, the system attempts an AX hit-test path first. It uses AXRaise, kAXMainAttribute, and kAXFocusedAttribute to interact with elements. Only if these fail does it fall back to NSRunningApplication.activate and global HID.
2. Targeted Keyboard Input: Instead of forcing application activation for every keystroke, type_text and press_key are implemented to deliver keyboard events directly to a specific Process ID (PID).
3. State Reading: get_app_state is designed to read application state without forcing the application to become the active/focused window.

Model-Side Preferences

To support this runtime behavior, ToolDefinitions and plugin manifests include 'Tool Intrusion Hints'. This encourages the model to prioritize low-side-effect tools in this order:

• get_app_state
• press_key
• type_text
• set_value
• perform_secondary_action

This approach aims to achieve a "keyboard-first, AX-first, global mouse last" behavior pattern.