DOCSEARCHbyAlgolia
Submit Docs

Unity CLI Loop

repository·main·Indexed 19 days ago

https://github.com/hatayama/unity-cli-loop

An AI-driven development loop for Unity (version 2022.3 or later) that enables AI agents to autonomously develop projects. It exposes Editor and PlayMode operations through a CLI (uloop-cli) and MCP-compatible skills, allowing LLM tools like Claude Code and Cursor to execute compilation, retrieve console logs, run tests, manage the scene hierarchy, and execute dynamic C# code.

Tokens
74.7K
Snippets
229
Records
304
Agent score
66%

What's inside unity-cli-loop

  1. What is Unity CLI Loop?

    main

    Unity CLI Loop is a tool designed to enable AI agents to autonomously perform development loops within Unity projects. It allows LLM tools (like Claude Code, Cursor, or GitHub Copilot) to execute tasks that humans typically do manually, such as compiling, running tests, analyzing logs, manipulating scenes, and verifying UI layouts via screenshots.

    Core capabilities include:

    1. Autonomous Development Loops: Automating compile, run-tests, get-logs, and clear-console.
    2. Editor Manipulation: Delegating scene construction, object manipulation, and menu execution via execute-dynamic-code and screenshot.
    3. PlayMode Automation: Simulating mouse/keyboard input, recording/replaying inputs, and verifying game behavior.
    4. Minimal Toolset: Achieving high autonomy with a streamlined set of commands.

  2. What is Unity CLI Loop and its core concepts

    main

    Unity CLI Loop is a tool designed to enable AI agents to autonomously perform compilation, testing, and manipulation within Unity projects. It allows LLM tools to bridge the gap between text-based instructions and Unity Editor operations.

    The core concepts are:

    1. Autonomous Development Loop: AI can independently run compile, run-tests, get-logs, and clear-console to fix errors.
    2. Editor Delegation: AI can delegate scene construction, object manipulation, menu execution, and UI layout verification via screenshots to the Unity Editor.
    3. PlayMode Automation: AI can execute automated tests during PlayMode, including simulating mouse/keyboard input, recording/replaying inputs, and verifying game behavior.
    4. Minimalist Toolset: Achieving high automation with a minimal number of core tools.

  3. Available Unity Skills

    main

    Skills are dynamically loaded from the uLoopMCP package in your Unity project. The following default skills are provided:

    • /uloop-compile - Execute compilation
    • /uloop-get-logs - Get console logs
    • /uloop-run-tests - Run tests
    • /uloop-clear-console - Clear console
    • /uloop-focus-window - Bring Unity Editor to front
    • /uloop-get-hierarchy - Get scene hierarchy
    • /uloop-find-game-objects - Find GameObjects
    • /uloop-screenshot - Take a screenshot of EditorWindow
    • /uloop-control-play-mode - Control Play Mode
    • /uloop-execute-dynamic-code - Execute dynamic C# code
    • /uloop-launch - Launch Unity project with matching Editor version

    Custom skills defined in your project are also automatically detected.

  4. When to use simulate-mouse-input vs simulate-mouse-ui

    main

    Choosing the correct tool depends on whether you are interacting with the Game World (Input System) or the Unity UI (EventSystem).

    ScenarioTool
    Click a Unity UI Button (IPointerClickHandler)simulate-mouse-ui
    Destroy a block in Minecraft (reads Mouse.current.leftButton)simulate-mouse-input
    Place a block with right-clicksimulate-mouse-input --button Right
    Drag a UI slidersimulate-mouse-ui --action Drag
    Look around with mouse (FPS camera)simulate-mouse-input --action MoveDelta
    Scroll hotbar slotssimulate-mouse-input --action Scroll

  5. How Skills work for custom tools

    main

    To make your custom tools automatically recognizable and usable by LLMs via the Skills system, you must provide a Skill/ definition within your tool's directory.

    Workflow:

    1. Create a Skill/ subfolder inside your tool's folder (e.g., Assets/Editor/CustomTools/MyTool/Skill/).
    2. Place a SKILL.md file inside that folder. This file is mandatory.
    3. Run uloop skills install --claude to bundle and install the project's Skills.

    Directory Structure:

    Assets/Editor/CustomTools/MyTool/
├── MyTool.cs           # Tool implementation
└── Skill/
    ├── SKILL.md        # Mandatory skill definition
    └── references/
        └── usage.md    # Optional additional files

    SKILL.md Format: The file must include a YAML frontmatter block with name and description.

    Scanning Locations:

    • Assets/**/Editor/<ToolFolder>/Skill/SKILL.md
    • Packages/*/Editor/<ToolFolder>/Skill/SKILL.md
    • Library/PackageCache/*/Editor/<ToolFolder>/Skill/SKILL.md

    Tips:

    • Add internal: true to the frontmatter to exclude the skill from installation (useful for debug tools).
    • Any files in Skill/ (like references/ or scripts/) are copied during installation.
    ---
name: uloop-my-custom-tool
description: "Description of the tool and when to use it"
---

# uloop my-custom-tool

Detailed tool documentation...

  6. Implement custom tools using the uloop-hello-world pattern

    main

    The uloop-hello-world skill serves as a reference implementation for creating custom tools within the uloop ecosystem. When building your own tools, follow these patterns demonstrated by this sample:

    • Type-safe parameter handling: Use Schema to define parameter types.
    • Enum parameters: Use enums for restricted selection (e.g., selecting a specific language from a list like english, japanese, spanish, french).
    • Boolean flags: Implement toggleable options (e.g., --include-timestamp).
    • Structured JSON output: Ensure the tool returns a consistent JSON schema containing the expected data fields.

  7. Understand the execute-dynamic-code tool contract

    main

    The execute-dynamic-code tool is designed with a stable public contract that remains consistent regardless of the underlying compiler strategy. When using this tool, you should expect the following behaviors:

    • Consistent Semantics: Whether the system uses the fast path (Shared Roslyn worker), a one-shot Roslyn compilation, or an AssemblyBuilder fallback, the functional behavior and the shape of the ExecuteDynamicCodeResponse remain identical. Fallbacks only affect performance, not the logic of the code being executed.
    • Security Guarantees: Security validation is an invariant. In restricted modes, the system performs both metadata and IL validation before any assembly is loaded via Assembly.Load.
    • Error Monitoring: If the system falls back from the preferred Roslyn worker to a slower compilation method, this is tracked as a health issue via the DynamicCompilationHealthMonitor.
    • Lifecycle: The compiler worker state is disposable. If a domain reload occurs or the worker protocol fails, the system is designed to rebuild state rather than attempting to preserve potentially corrupted state.

  8. How the Dynamic Code Execution workflow works

    main

    The execution of dynamic code follows a structured path through the system layers:

    1. Trigger: An external tool calls ExecuteDynamicCodeTool.
    2. Orchestration: ExecuteDynamicCodeUseCase resolves the security level, converts parameters, and manages retries.
    3. Runtime Gateway: The UseCase interacts with IDynamicCodeExecutionRuntime (implemented by DynamicCodeExecutionFacade).
    4. Compilation: The DynamicCodeCompiler coordinates:
      • Cache lookup.
      • Source preparation via DynamicCodeSourcePreparer (handling literal hoisting and wrapper generation).
      • Planning via DynamicCompilationPlanner.
      • Building via CompiledAssemblyBuilder (which uses DynamicCompilationBackend).
      • Loading via CompiledAssemblyLoader.
    5. Execution: The DynamicCodeExecutor bridges the gap by invoking the compiled wrapper through the CommandRunner and CompiledCommandEntryPointResolver.

  9. How Unity CLI Loop is designed

    main

    The design philosophy of Unity CLI Loop prioritizes a minimal toolset to prevent AI confusion and context window bloat.

    Instead of providing hundreds of specific tools, it relies on execute-dynamic-code (C# dynamic execution) to handle most Unity Editor operations. Dedicated tools are only provided for:

    1. Operations that cannot be handled by dynamic code: e.g., input simulation across frames or taking screenshots.
    2. High-frequency development loop tasks: e.g., compile and get-logs. These are specialized to reduce token costs by avoiding the need to generate C# code for every single call.

  10. Choose between execute-dynamic-code and mouse simulation tools

    main

    When automating Unity PlayMode, choose your tool based on whether you need to test the input pipeline or just the resulting state.

    ScenarioRecommended toolWhy
    Verify uGUI element response via EventSystemsimulate-mouse-uiFires PointerDown, PointerUp, PointerClick, and drag events through raycasts.
    Test gameplay reading Mouse.current (New Input System)simulate-mouse-inputInjects state into Mouse.current (e.g., wasPressedThisFrame, delta).
    Jump to button callbacks, invoke methods, or set preconditionsexecute-dynamic-codeBest for direct automation without the input pipeline.
    Drive custom runtime behavior not mapped to mouse toolsexecute-dynamic-codeAllows calling project-specific methods and inspecting scene objects.

  11. Understand the execute-dynamic-code pipeline architecture

    main

    The execute-dynamic-code pipeline is organized into a layered architecture designed to separate external entry points from low-level execution mechanics. The architecture follows a strict Entry -> UseCase -> Infrastructure flow.

    Layered Responsibilities

    1. Entry Layer: Translates external calls (from MCP or CLI) into use-case invocations. It contains ExecuteDynamicCodeTool and McpServerController. It should not contain business logic or direct wiring to compilers.
    2. UseCase Layer: Manages the temporal cohesion and workflow order. It handles user-facing logic like retry rules (e.g., the missing-return retry). Key components are ExecuteDynamicCodeUseCase and PrewarmDynamicCodeUseCase.
    3. Infrastructure Layer: Contains the heavy mechanics. It is subdivided into functional modules:
      • Runtime Access: The gateway to the infrastructure, exposing IDynamicCodeExecutionRuntime to the UseCase layer.
      • Planning: Converts requests into a DynamicCompilationPlan.
      • Backend Build: Produces CompiledAssemblyBuildResult from a plan, handling reference resolution and backend selection (Roslyn vs. Fallback).
      • Safety + Load: Handles assembly loading, metadata validation, and IL validation.
      • Invocation: Executes the compiled code via ICompiledCommandInvoker.
      • Compilation Pipeline: Orchestrates the entire process via DynamicCodeCompiler.

  12. Coordinate system for mouse input simulation

    main

    The uloop simulate-mouse-input tool uses a top-left Game View coordinate system for --x and --y parameters.

    • Input Coordinates: --x and --y are relative to the top-left of the Game View.
    • Internal Conversion: The tool automatically converts these to Unity's internal Mouse.current.position (which uses bottom-left coordinates) using the formula:
      • unity_x = input_x
      • unity_y = gameViewHeight - input_y

    Note: Do not manually flip the Y coordinate in your command; the tool handles this conversion internally.