TuriX Computer-Use Agent (CUA)

repository·main·Indexed 25 days ago

https://github.com/turixai/turix-cua

An open-source computer-use agent (CUA) that enables AI models to perform desktop automation on macOS, Windows, and Linux by interacting with the GUI. It supports custom model integration via config.json and Model Context Protocol (MCP), and integrates with OpenClaw and Clawdbot. The system allows for high-level task decomposition via a Planner, the use of markdown-based skills for domain-specific guidance, and supports resuming interrupted tasks.

Tokens
13.8K
Snippets
22
Records
98
Agent score
84%

What's inside TuriX-CUA

  1. Overview of TuriX Key Features

    main

    TuriX is an AI-driven desktop automation agent with the following core capabilities:

    • SOTA Default Models: High success rates and speed on macOS, outperforming previous open-source agents like UI-TARS.
    • No App-Specific APIs Required: Operates via GUI interaction (clicks/typing), meaning it can use any application a human can (WhatsApp, Excel, Outlook, etc.).
    • Hot-swappable 'Brain': Easily replace VLM strategies via config.json.
    • MCP Ready: Can be integrated with Claude for Desktop or any agent supporting the Model Context Protocol (MCP).
    • Skills (Markdown Manuals): The Planner selects skills based on names/descriptions, while the Brain uses the full skill content to guide execution steps.
  2. Quickstart for TuriX on macOS 15+

    main

    To set up TuriX on macOS 15+, follow these steps:

    1. Download the App: Obtain the application from the official website.
    2. Create a Python 3.12 Environment: Ensure you have a dedicated Python 3.12 environment prepared.
    3. Grant macOS Permissions: You must grant specific permissions for the agent to operate:
      • Accessibility: Required for controlling the UI.
      • Safari Automation: Required if the agent needs to interact with Safari.
    4. Configure and Run: Set up your config.json and execute the application.
    5. Skills (Optional): You can optionally add skills to enhance capabilities.
  3. Use TuriX with OpenClaw Skills

    main

    You can use TuriX via OpenClaw using the published ClawHub skill. The repository also contains local OpenClaw skill packages for different operating systems:

    • macOS: Located in the main branch. Includes SKILL.md and scripts/run_turix.sh.
    • Windows: Located in the multi-agent-windows branch. Includes SKILL.md, scripts/run_turix.ps1, and agents/openai.yaml.

    For specific installation and permission instructions for OpenClaw, refer to OpenCLaw_TuriX_skill/README.md within the repository.

  4. Create a custom TuriX Skill

    main

    Skills are markdown playbooks that guide the agent through specific domains. To create a new skill, create a .md file in the skills/ directory with YAML frontmatter.

    • name: The identifier used by the Planner to select the skill.
    • description: A description that the Planner uses to match the current task to this skill.
    • Body: The markdown content used by the Brain to provide step-by-step execution guidance.

    Example skills/browser-tasks.md:

    ---
    name: browser-tasks
    description: When performing tasks in a web browser (search, navigate, fill forms).
    ---
    # Browser Tasks
    
    ## Navigation
    - Use the address bar or search box to navigate
    - Open new tabs for each distinct task
    - Wait for page to fully load before proceeding
    
    ## Forms
    - Click on input fields to focus
    - Type content clearly
    - Look for submit/button to complete actions
    
    ## Safety
    - Confirm before submitting forms
    - Do not download files without user permission
    ---
    name: my-custom-skill
    description: When performing X specific task
    ---
    # Custom Skill
    
    ## Guidelines
    - Step 1: Do this first
    - Step 2: Then do that
    - Step 3: Verify the result
  5. Resume interrupted tasks

    main

    If a task is interrupted, you can resume from the last position by configuring a fixed agent_id and enabling resume in examples/config.json.

    Requirements for Resuming:

    • Use the same agent_id used in the previous run.
    • Keep the same task string.
    • A memory file must exist at src/agent/temp_files/<agent_id>/memory.jsonl.

    To start fresh, set resume to false, change the agent_id, or delete the directory src/agent/temp_files/<agent_id>.

    {
        "agent": {
             "resume": true,
             "agent_id": "my-task-001"
        }
    }
  6. Use Skills to extend Agent capabilities

    main

    Skills are Markdown files located in a directory (default skills/) that provide instructions for specific tasks.

    1. Format: Each file must start with YAML frontmatter containing name and description.
    2. Usage: The Planner uses the name/description to select the skill, while the Brain uses the full content for step-by-step guidance.
    3. Requirement: You must enable planning (agent.use_plan: true) to use skills.

    Example Skill File (skills/github-web-actions.md):

    ---
    name: github-web-actions
    description: 用于在浏览器中操作 GitHub(搜索仓库、点 Star 等)。
    ---
    # GitHub Web Actions
    - 打开 GitHub,使用站内搜索并进入仓库页面。
    - 若需要登录,先向用户确认再继续。
    - 在继续之前确认 Star 按钮状态。

    Enable in examples/config.json:

    {
      "agent": {
        "use_plan": true,
        "use_skills": true,
        "skills_dir": "skills",
        "skills_max_chars": 4000
      }
    }
  7. Install and Setup TuriX-Mac Clawdbot Skill

    main

    To enable Clawdbot to control your macOS desktop visually, you must set up the TuriX Core and configure the skill directory.

    1. TuriX Core Setup: Install the core requirements from the official TuriX-CUA repository:
    conda activate your_env
    pip install -r requirements.txt
    1. Skill Directory Structure: Place the skill files in your local Clawdbot skills directory following this structure:
    your_dir/clawd/skills/local/turix-mac/
    ├── README.md
    ├── SKILL.md
    └── scripts/
        └── run_turix.sh
    conda activate your_env
    pip install -r requirements.txt