OpenAI Testing Agent Demo

The Testing Agent Demo is a monorepo demonstrating how to automate frontend testing using OpenAI's CUA model and the computer use tool. It uses Playwright to drive a browser instance.

The system consists of three interconnected components:

1. frontend: A Next.js web interface used to configure test cases and monitor their execution.
2. cua-server: A Node.js service that acts as the orchestrator. It communicates with the OpenAI CUA model and uses Playwright to interact with the target web application.
3. sample-test-app: An example e-commerce site used as the target for testing demonstrations.

The CUA Server is a Node.js service that interfaces with the OpenAI CUA model and provides a Socket.IO WebSocket API for the frontend.

To set up the server:

1. Create a development environment file from the example:

   cp .env.example .env.development

2. Open .env.development and add your OPENAI_API_KEY.
3. Install dependencies and the Playwright browser binaries:

   npm install
   npx playwright install

4. Start the server:

   npm run dev
   # or
   npm start

The server listens on port 8000 by default.

To use the testing agent with a different web application, you can modify the target URL and the test case. This can be done in two ways:

1. Via the UI: Use the configuration options provided in the frontend web interface.
2. Via Code: Update the default values in frontend/lib/constants.ts.

If you want to integrate the testing agent into your own project, focus on the cua-server package, as it contains the core logic for communicating with the CUA model and driving Playwright.

Before starting the development server, you must configure your environment variables. The application uses .env.development for local development.

Specifically, you must provide an OpenAI key in this file to enable the agent's capabilities.

cp .env.example .env.development
# Edit .env.development to include your OpenAI key

The Sample Test App is a Next.js 14 e-commerce demo used as a target for the automated testing agent. To set up the local development environment:

1. Configure Environment Variables: Copy the example environment file to .env.development and optionally update ADMIN_USERNAME and ADMIN_PASSWORD.
2. Sync Credentials: If you change the admin credentials in the .env.development file, you must also update them in the frontend configuration (the default values are located in frontend/lib/constants.ts).
3. Install and Run: Install dependencies via npm and start the development server.

The application runs at http://localhost:3005.

# 1. Copy environment file
cp .env.example .env.development

# 2. Install dependencies and start dev server
npm install
npm run dev

The Frontend UI is a Next.js application that serves as the web interface for the Testing Agent demo. It enables users to define test cases, transmit them to the CUA server, and visualize messages from the CUA model.

To run the frontend, you must ensure the CUA server is running and, for a complete demo experience, the sample-test-app is also active.

Prerequisites

• A running CUA server.
• A running sample-test-app (optional, for demo purposes).

# 1. Prepare environment variables
cp .env.example .env.development
# (Manually edit .env.development to add your OpenAI key)

# 2. Install dependencies
npm install

# 3. Start the development server
npm run dev

Caution

Computer use is currently in preview. Because the model is in preview, it may be susceptible to exploits or inadvertent mistakes.

Best Practices:

• Use this project in test environments only.
• Do not use real user data in production.
• Avoid trusting the agent in authenticated environments or for high-stakes tasks.

The sample-test-app uses environment variables to define administrative credentials. These credentials are used to authenticate administrative actions within the application during testing. Set these in your .env file based on the following keys:

ADMIN_USERNAME=test_user_name
ADMIN_PASSWORD=test_password

The frontend requires an OpenAI API key to function. You can also optionally configure the browser's viewport dimensions and provide environment-specific instructions to guide the testing agent's behavior (e.g., specifying the operating system or keyboard modifier keys).

OPENAI_API_KEY=your-openai-key

# Optional: Set the display size for the browser.
DISPLAY_WIDTH=1024
DISPLAY_HEIGHT=768

# Optional: Add any environment specific instructions here.
ENV_SPECIFIC_INSTRUCTIONS="This is MacOS, so use CMD key for actions that are typically done with Control key on Windows."

The cuaLoopHandler determines the browser viewport size using the following environment variables. If these are not set, the handler defaults to 1024x768.

• DISPLAY_WIDTH: The width of the browser viewport (integer).
• DISPLAY_HEIGHT: The height of the browser viewport (integer).

# Example setting dimensions in a shell environment
export DISPLAY_WIDTH=1920
export DISPLAY_HEIGHT=1080

The cua-server uses environment variables to configure its network settings and CORS policy.

• SOCKET_PORT: The port on which the Socket.IO server will listen. Defaults to 8000 if not provided.
• CORS_ORIGIN: The allowed origin for Cross-Origin Resource Sharing. Defaults to * (all origins).