passmark

CUA mode (mode: "cua") uses OpenAI's Responses API directly with the computer tool. This mode has specific requirements:

• Gateway Restriction: CUA does not work through gateways (Vercel, OpenRouter, Cloudflare). You must set gateway: "none" in your configuration.
• Required Key: OPENAI_API_KEY must be set in your environment.
• Access Issues: If you receive a 400 error with param: null, verify your OpenAI organization has access to the CUA model and the computer tool on the Responses API.
• Tool Compatibility: If you see errors regarding computer_use_preview, you are likely on an old build. The current API uses { type: "computer" }. Rebuild from main to resolve.

Passmark provides a placeholder system to inject dynamic values into step data. These can be scoped to a single test or shared globally across an execution.

By default, Passmark uses ARIA accessibility snapshots. For visual, screenshot-driven automation using OpenAI's computer-use agent, configure the mode to "cua".

Requirements & Limitations:

• Requires gateway: "none" (CUA requires direct OpenAI access).
• Requires OPENAI_API_KEY in your .env.
• The CUA model (gpt-5.5 + computer tool) is currently locked and not user-configurable.
• Redis step caching is skipped in CUA mode.
• Not compatible with vercel, openrouter, or cloudflare gateways.

import { configure } from "passmark";

configure({
  ai: {
    mode: "cua",
    gateway: "none", // CUA requires direct OpenAI access
  },
});

Passmark uses Redis to cache successful step actions. On subsequent runs, cached steps execute directly without AI calls, reducing latency and cost.

Key Details:

• Steps are cached by userFlow + step.description.
• To force AI execution, set bypassCache: true on individual steps or the entire run.
• Cache is automatically bypassed on Playwright retries.
• Limitation: Currently, only single-step AI executions are cached; multi-step actions are not yet cached.

Call configure() once before executing any core functions to set up AI gateways, models, and upload paths.

import { configure } from "passmark";

configure({
  ai: {
    gateway: "none", // "none" (default), "vercel", "openrouter", "opencodezen", or "cloudflare"
    models: {
      stepExecution: "google/gemini-3-flash",
      utility: "google/gemini-2.5-flash",
    },
  },
  uploadBasePath: "./uploads",
});

To use Passmark, initialize a Playwright project and install the passmark package. It is recommended to use TypeScript.

npm init playwright@latest passmark-project # select the default options and set language to TypeScript
cd passmark-project
npm install passmark

To test flows involving email verification, configure an email provider. You can use the built-in emailsink provider or implement a custom one.

Using emailsink:

import { configure } from "passmark";
import { emailsinkProvider } from "passmark/providers/emailsink";

configure({
  email: emailsinkProvider({ apiKey: process.env.EMAILSINK_API_KEY }),
});

Custom Provider Example:

configure({
  email: {
    domain: "my-test-mail.com",
    extractContent: async ({ email, prompt }) => {
      // Fetch and extract content from your email service
      return extractedValue;
    },
  },
});

Usage in steps: Use the {{email.type:prompt}} pattern to extract data:

{ 
  description: "Enter the verification code", 
  data: { value: "{{email.otp:get the 6 digit verification code:{{run.dynamicEmail}}}}" } 
}

To ensure your Playwright tests can read the .env file containing your API keys, add the following to your playwright.config.ts file. You must first install dotenv via npm install dotenv.

import dotenv from 'dotenv';
import path from 'path';

dotenv.config({ path: path.resolve(__dirname, '.env') });

To get more detailed information from the Passmark logger, set the PASSMARK_LOG_LEVEL environment variable to debug before running your commands.

export PASSMARK_LOG_LEVEL=debug
pnpm test

Before running Passmark, ensure your environment meets these requirements:

• Node.js: version 18 or higher.
• Dependencies: Install via pnpm install (or npm install).
• Browsers: Install Playwright browsers using npx playwright install.
• Redis: Must be available at the REDIS_URL or configured via configure({ redis: { url } }).
• AI API Keys:
  • For direct providers: ANTHROPIC_API_KEY or GOOGLE_GENERATIVE_AI_API_KEY.
  • For Vercel AI Gateway: AI_GATEWAY_API_KEY.
  • For CUA mode (mode: "cua"): OPENAI_API_KEY and gateway: "none".

Passmark requires API keys to call model providers. You can set these via environment variables or through the configure function.

Direct Provider Usage

Set the following environment variables:

export ANTHROPIC_API_KEY=sk-...
export GOOGLE_GENERATIVE_AI_API_KEY=AIza...

Vercel AI Gateway

To use the Vercel AI Gateway, call configure({ ai: { gateway: 'vercel' } }) and set the required key:

export AI_GATEWAY_API_KEY=your_gateway_key