openai-cua-sample-app

The Booking Lab Template is a run-scoped HTTP commerce lab designed to demonstrate specific agentic workflows. It provides the infrastructure for an agent to perform the following tasks:

1. Filter inventory: Querying available products or services.
2. Complete a reservation form: Interacting with commerce interfaces to submit data.
3. Verify local confirmation: Checking that a reservation record was successfully created locally.

The primary hero scenario implemented in this template is booking-complete-reservation.

When running scenarios, you can choose between two execution modes. The verification process remains identical for both because it inspects the final state of the lab rather than the agent's transcript.

• code mode: Uses the browser REPL tool (exec_js) to drive the lab.
• native mode: Uses the computer tool directly to interact with the environment.

The execution lifecycle follows this sequence:

1. Discovery: The operator console (demo-web) requests the public scenario registry from the runner.
2. Initialization: Starting a run triggers RunnerManager to create a mutable workspace and a replay bundle.
3. Selection: RunnerManager selects the appropriate scenario executor via executor-registry.ts.
4. Execution: The executor launches the lab and hands control to responses-loop.ts.
5. Feedback Loop: The loop emits events, screenshots, and final verification results into the replay bundle.
6. Observation: The web app reads run details and consumes SSE (Server-Sent Events) updates until the run completes.

The sample app supports two distinct modes for interacting with the browser lab:

• native: This mode exposes the Responses API computer tool directly. The model performs high-level computer actions like clicks, drags, typing, waits, and taking screenshots against a live browser session.
• code: This mode exposes a persistent Playwright JavaScript REPL via the exec_js tool. Instead of raw computer actions, the model writes and executes JavaScript scripts to control the browser.

Both modes utilize the same scenario manifests and the replay pipeline.

To add a new scenario, follow these steps across the package boundaries to ensure proper integration with the runner and UI:

1. Create the lab template in labs/<name>-lab-template.
2. Define the scenario manifest and default prompt in packages/scenario-kit/src.
3. Export the manifest via packages/scenario-kit/src/scenarios.ts.
4. Implement scenario instructions and verification helpers in packages/runner-core/src.
5. Register the executor in packages/runner-core/src/executor-registry.ts.
6. Update or add tests covering the manifest, runner behavior, and UI guidance.

If you need to view independent logs for the runner and the web console, you can start them in separate terminal sessions:

• Runner: Use pnpm dev:runner.
• Web App: Use RUNNER_BASE_URL=http://127.0.0.1:4001 pnpm dev:web (ensure the URL matches your runner's address).

Before submitting changes, run the standard quality gate commands. If you need to run live smoke tests against the Responses API, ensure you have an OPENAI_API_KEY configured and use the pnpm test:live command.

pnpm lint
pnpm typecheck
pnpm test
pnpm build

Lab templates must be self-contained and resettable. Follow these guidelines:

• Asset Management: Keep all assets local to the specific template folder.
• Verification: Expose stable browser-side accessors to facilitate verification.
• Dependencies: Avoid network dependencies unless they are essential to the scenario story.
• Observability: Ensure the initial state is easy to reason about from screenshots.