DOCSEARCHbyAlgolia
Submit Docs

FinalRun Agent

repository·main·Indexed 43 days ago

https://github.com/droid-ash/finalrun-agent

An AI-driven CLI tool for testing Android and iOS applications using natural language. It translates plain-English YAML test specs into physical device interactions. FinalRun supports a 'Bring Your Own Key' (BYOK) model for Gemini, GPT, and Claude, and provides specialized AI coding agent skills for generating, running, and fixing tests via a full automate-diagnose-fix loop.

Tokens
130.7K
Snippets
282
Records
590
Agent score
55%

What's inside FinalRun

  1. Introduction to FinalRun AI-powered test automation

    main

    FinalRun is an AI-powered automation platform for Android and iOS apps. It allows you to describe test scenarios in plain English, which are then converted into YAML test cases using AI agent skills (compatible with Claude Code, Cursor, and Codex).

    Core Workflow:

    1. Scenario Definition: Describe tests in natural language.
    2. Test Generation: AI agents transform descriptions into YAML test cases.
    3. Execution: FinalRun launches the app on a real device, emulator, or simulator.
    4. AI-Driven Interaction: An AI model interprets the screen to perform actions like tapping, swiping, typing, and verifying results.
    5. Reporting: Generates pass/fail reports including video recordings, screenshots, and device logs.

  2. Overview of @finalrun/report-web

    main

    The report-web package is a Vite React Single Page Application (SPA) designed to view test run results. It also functions as an importable UI library.

    Key architectural characteristics:

    • Logic Separation: View-model, log, format, and route logic are located under src/ui and src/.
    • Stability: Core logic is protected by characterization tests.
    • Input Contract: The published @finalrun/report-web/ui barrel relies on its declared types for its input contract rather than using defensive guards.
    • Device Log Viewing: The device-log viewer component is DeviceLogPanel.tsx.
    • Data Loading: Manifest loading and the 500-line device-log tail logic are handled in packages/cli/src/reportViewModel.ts.

  3. Architecture Overview of FinalRun

    main

    FinalRun is an AI-powered mobile app testing tool organized as a monorepo with 7 specialized packages. This separation allows for a modular architecture where the AI execution loop, device management, and user-facing CLI are decoupled.

    Package Responsibilities

    • common: Shared TypeScript types and contracts.
    • cli: User-facing orchestration, I/O, and report server.
    • goal-executor: The AI agent loop (screenshot $\rightarrow$ LLM plan $\rightarrow$ device action $\rightarrow$ repeat).
    • device-node: Platform-specific (Android/iOS) device management via gRPC.
    • cloud-core: Logic for cloud submission.
    • local-runtime: Per-platform asset bundles (driver APKs/iOS zips, proto, report SPA).
    • report-web: React SPA used for viewing test reports.

  4. Understand the FinalRun Package Architecture

    main

    The FinalRun project is organized into several specialized packages that handle different stages of the test lifecycle, from configuration parsing to AI-driven execution and reporting:

    • @finalrun/common: Contains the core data models (contracts for runs, tests, results, and device actions), YAML parsers (testLoader.ts), and workspace discovery logic.
    • @finalrun/cli: The entry point for users. It handles command dispatching, test orchestration, AI prompt compilation, and manages the built-in report server.
    • @finalrun/goal-executor: The intelligence engine. It implements the AI loop (capture → plan → act → repeat) using TestExecutor.ts and manages LLM interactions via AIAgent.ts.
    • @finalrun/device-node: Manages physical and virtual hardware. It includes device managers (DeviceNode.ts), Android/iOS runtimes, and gRPC drivers for communicating with devices.
    • @finalrun/report-web: A Next.js-based web application that renders HTML reports and serves test artifacts (screenshots, JSON traces) for visual debugging.

  5. Available FinalRun AI agent skills

    main

    FinalRun provides three primary skills for automating mobile test lifecycles via chat:

    • /finalrun-generate-test: Reads source code to infer app identity and generates YAML test specs organized by feature folder.
    • /finalrun-use-cli: Validates and executes generated tests, handling CLI flags, environment bindings, and artifact inspection (logs, screenshots, result.json).
    • /finalrun-test-and-fix: An orchestration skill that automates the full generate → run → diagnose → fix loop until tests pass or a blocker is hit.

  6. Understand the CLI orchestration components

    main

    The Finalrun CLI is composed of four primary functional modules that manage the lifecycle of an AI-driven test session:

    1. host-preflight: Performs environment checks before a session starts. It uses platform-specific probes (where on Windows, which on other platforms) via resolveCommandPath to ensure required commands are available on the system PATH.
    2. report-writer: Manages the output directory for a run. It generates runner.log, input/, tests/<id>/, summary.json, and run.json. It enforces a secret-redaction contract across all write paths and follows a first-failure precedence rule.
    3. session-runner: Orchestrates the execution lifecycle via prepareTestSession, executeTestOnSession, and runGoal. It manages the ExecutionSessionState, handles device-log capture, and ensures all held resources are released in a single finally block.
    4. test-compiler: Responsible for compileTestObjective. It handles variable substitution (only ${variables.*} is substituted at compile-time) to ensure compile-time guarantees for the objective. It anchors the secret-value exposure chain, ensuring secrets reach providers through captured hierarchies while redacting prompt text at the AIAgent seam.

  7. How Validation (runCheck) Works

    main

    Before any device interaction occurs, the runCheck phase (implemented in packages/common/src/checkRunner.ts) performs several validation steps to ensure the test run will not fail due to configuration errors:

    1. Target Resolution: Resolves CLI selectors or suite manifests into absolute file paths.
    2. Test Loading: Parses YAML files and validates required fields (name, steps).
    3. Binding Validation: Scans for ${variables.*} and ${secrets.*} references and verifies they exist in the resolved environment config. If a binding is missing, the process throws an error immediately.
    4. App Configuration: Merges environment overrides with workspace config to determine the target platform (Android via packageName or iOS via bundleId).

  8. Understand the Comment Deletion Test (Restatement vs. Rationale)

    main

    When auditing or cleaning up comments in packages/ and drivers/ (TypeScript, Kotlin, Swift), use the per-claim deletion test to distinguish between 'restatement' and 'rationale'.

    • Restatement (Remove): A comment is restatement if a competent reader can recover everything it said from the code itself. For example, in JSDoc/KDoc, a line like @param name - the name is restatement and should be removed.
    • Rationale (Keep): Comments that explain why a decision was made, non-obvious constraints, cross-file couplings, references to external systems (ruleset IDs, tickets, vendor behavior), or measured data must be kept.

    Rule of Thumb: If a block contains both, edit it to remove the restatement sentences while preserving the rationale sentences. If a comment is ambiguous, keep it to avoid losing unrecoverable context.

  9. Determine log capture stop outcome from stream terminal state

    main

    The success or failure of a stopLogCapture operation is determined by the terminal state of the write stream, not just the presence of an error event.

    To ensure durability, finalize must wait for the stream to reach the 'close' state. A stop is considered successful only if stream.writableFinished is true and stream.errored is null.

    Why wait for 'close'? An error (like EIO) can occur during the close(2) syscall after the stream has successfully finished flushing data. Deciding the outcome before 'close' lands creates a race condition where a stop might be reported as successful even if the OS refused to confirm the file's durability.

  10. Understand LogWriteStream finalization behavior

    main

    The finalize method in packages/device-node/src/device/logWriteStream.ts determines the success or failure of a log stream based on its terminal state.

    Success Criteria: finalize resolves if writableFinished is true AND stream.errored is null.

    Error Handling:

    • If a stale error was recorded (an error exists but the stream is otherwise healthy), finalize will resolve but emit a warning via Logger.w naming the file and the error.
    • In all other error cases, finalize rejects with entry.error ?? stream.errored ?? new Error(...).
    • A 'close-time error' (an error occurring during the stream's destruction phase) will cause finalize to reject deterministically, even if writableFinished is true.

    Key Invariant: finalize waits for the stream's 'close' event using a never-rejecting listener to ensure it reaches a true terminal state before making a decision. This prevents the method from hanging or throwing prematurely during the teardown process.

  11. How `finalize` handles terminal stream states

    main

    To prevent race conditions where errors occurring during the file close process are missed, finalize (specifically within _endAndFlush in packages/device-node/src/device/logWriteStream.ts) now awaits the terminal 'close' event of the stream.

    This ensures that every error the stream will ever deliver is recorded before the final decision to resolve or reject is made. The implementation uses a state-based wait (e.g., if (!stream.closed) await once(stream, 'close')) rather than stream/promises' finished() to avoid premature rejection on already-destroyed streams.

  12. How Secret Redaction works in artifacts

    main
    To prevent sensitive information from appearing in logs or HTML reports, the reportWriter performs automatic redaction. It uses the RuntimeBindings.secrets map to find and replace secret values in all written artifacts. For example, an input like "Entered password123 in field" is transformed into "Entered [REDACTED] in field" before being saved to disk.