DOCSEARCHbyAlgolia
Submit Docs

Playwright Reports Server

repository·main·Indexed 42 days ago

https://github.com/cyborgtests/playwright-reports-server

A system to accept, merge, store, and view Playwright reports. Includes a reporter package for uploading results and the pwrs-cli, a command-line interface that provides compact, JSON-formatted data optimized for AI coding agents to perform automated test analysis, debugging, and reporting.

Tokens
66.6K
Snippets
181
Records
292
Agent score
53%

What's inside Playwright Reports Server

  1. Overview of Playwright Reports Server features

    main

    Playwright Reports Server is a self-hosted solution for managing Playwright test results. Key capabilities include:

    • Report Management: Centralized storage for HTML reports, sharded run merging (server-side), and GitHub Actions artifact syncing.
    • Analytics: Dashboards for pass-rate trends, slow tests, failure categories, and failure clusters.
    • Test Stability: Flakiness scoring and test quarantine features.
    • AI Integration: LLM-powered failure analysis using OpenAI-compatible or Anthropic-format providers.
    • Extensibility: Pluggable storage (Local, S3, Azure Blob), granular cleanup rules, and notification support (Slack/Webhooks).
    • Security: Optional authentication (off by default) with roles (admin, reader, readonly), SSO (GitHub, Google, OIDC), and scoped API keys.

  2. Explore Playwright Reports Server documentation

    main

    The documentation is organized into several functional areas to help you manage your Playwright report infrastructure:

    Ops & Infrastructure

    • Configuration: Environment variables for system setup.
    • Authentication: Managing users, roles, invites, API keys, and SSO.
    • Storage: Configuring backends like fs, s3, or azure, including Litestream replication.
    • Cleanup: Setting retention rules for reports and artifacts.
    • Deployment: Docker-based deployment guidance.
    • Notifications: Sending report outcomes to Slack or webhooks via event rules (on upload) or schedule rules (cron).
    • UI White-label: Customizing the title, logo, favicon, and header links.

    Getting Reports In

    • Uploading Reports: Methods include the reporter plugin, sharded run auto-merging, CLI HTML uploads, GitHub Sync, manual generation, and direct-to-S3 presigned uploads.
    • Data Migration: Importing existing reports and results from another instance.
    • GitHub Sync: Automatically scraping Playwright artifacts from GitHub Actions via cron.

    Day-to-Day Features

    • Overview Dashboard: A configurable home page (/) with project letter grades and OK/NOT OK verdicts.
    • Analytics Dashboard: Viewing trends, slow tests, and failure clustering.
    • Regression Tracking: Distinguishing genuine breaks from flakes by tracking breaking/last-green commits.
    • Test Management & Quarantine: Scoring flakiness and using auto-quarantine (implemented via a file on disk).
    • Report Export (PDF): Generating server-side PDFs for audits.

    LLM Integration

    • Analysis: Failure analysis using providers, model registries, screenshots, and background queues.
    • Routing: Multi-model strategies (fusion, council, cascade, refine).
    • Selection: Guidance on choosing between local models (LM Studio, oMLX, Ollama) and remote models (OpenRouter).
    • Code Assistant: Using pwrs-cli with the Claude Code skill to provide AI agents with real run history.

  4. Interpret Cluster Confidence Levels

    main

    Every failure cluster is assigned a confidence level (high, medium, or low) indicating how reliably a single fix will resolve all tests in that cluster.

    ConfidenceMeaning
    highA fixture cluster, OR $\ge 3$ tests share the same frame or selector anchor.
    medium2 tests share an anchor, OR one test fails chronically ($\ge 3$ times) at the same anchor.
    lowA single-test single-failure, or an unmatched failure.

  5. Use the Fusion Routing Strategy

    main

    The Fusion strategy uses several authors to draft the same task in parallel, which are then merged by a synthesizer into a single answer.

    • Lenses: Each author can be assigned a lens (a specific point of view, e.g., assume the app is at fault vs be a skeptic). This provides variety even when using the same model.
    • Pros: Can improve quality by combining different perspectives and resolving contradictions.
    • Cons: Only effective if authors are genuinely different (different families or different lenses). Identical authors with no lenses share the same blind spots.
    • Best for: Report summary and Project summary.
    Settings -> LLM Configuration -> Routing -> Fusion

  6. Use special resultDetails keys for metadata and filtering

    main

    The resultDetails object allows you to attach metadata to your test runs. While most keys are treated as custom filterable tags, specific keys drive built-in server functionality:

    KeyEffect
    projectGroups reports and results. Used for Project filters, Quality Overview, analytics, and quarantine lookup (skipQuarantinedTests).
    environmentDeployment target (qa, staging, prod). Drives the Env column and the ?environment=qa filter.
    testRunRun ID used for sharded merges. Must be identical across all shards of a single run.
    branchDisplayed as a branch chip on the reports list and filterable as a tag.
    shardCurrentAuto-filled from Playwright's --shard=N/M (N). Do not set manually.
    shardTotalAuto-filled from Playwright's --shard=N/M (M). The server waits for this many blobs with the same testRun and project before merging.

    Note: The reporter also automatically injects playwrightVersion and username (from git user.name or QA_USERNAME). Custom keys (e.g., appVersion) are available as tags via ?tags=key:value.

  7. Criteria for choosing an LLM for failure analysis

    main

    When selecting a model for Playwright Reports Server failure analysis, evaluate it based on three primary dimensions rather than just chat performance:

    1. Visual Understanding (Screenshots): Test analysis relies on failure screenshots. Choose a vision-capable model. If using a text-only model, you must point a dedicated vision model at the transcriptions.
    2. Output Format: The system requires structured markdown. Avoid models that deviate from requested formats, as this breaks readability and reuse.
    3. Reasoning Efficiency: Avoid models that enter long, unproductive reasoning loops. A model with higher tokens-per-second (TPS) that answers quickly is often better than a larger model that overthinks. Judge performance by the total round-trip time on a real failure rather than raw TPS.

  8. Use the One-shot Routing Strategy

    main

    The One-shot strategy is the default and is recommended for almost all users. A single model produces the answer (the Primary model by default).

    • How to customize: You can pin a specific non-Primary model to a task using the model picker on that task's card. Pinned models bypass the fallback chain.
    • Pros: Cheapest, fastest, and simplest. A single strong model often outperforms an ensemble of smaller models for hard reasoning.
    • Use when: Always, unless you have a measured reason to use a multi-model strategy.
    Settings -> LLM Configuration -> Routing -> One-shot

  9. Understand the 'brief' payload concept

    main

    The test brief and report brief commands are the primary entry points for consuming data efficiently. They are designed to be compact (optimized for LLM context budgets) by performing multiple backend operations in a single HTTP call.

    test brief

    Provides a comprehensive view of a single test, including:

    • Signals: Quarantine state, flakiness score, and flaky tier.
    • Latest Failure: Error message, category, signature, location, and links to attachments (screenshots/error context).
    • LLM Analysis: Pre-computed root cause and fix.
    • Feedback: Human-provided notes.
    • Regression Data: Information on when a regression started and its impact.
    • Cluster Membership: If the test is part of a failure cluster.

    Note on Attachments: screenshotUrl, errorContextUrl, and reportUrl are server-relative paths. To fetch them, prepend the $PWRS_SERVER_URL and include Authorization: Bearer $PWRS_API_TOKEN.

    report brief

    Returns a discriminated union based on the mode key:

    • summary mode (default): A compact view (~5 KB) containing stats, clusterSummary, and sampleUnclusteredFailures.
    • full mode (via --with-failures): A detailed view (~100 KB) containing all summary data plus a failedTests array containing a TestBrief for every failure.

  10. Identify the fix target using Cluster Anchors

    main

    When tests are grouped into a cluster, the cluster.anchor identifies the deterministic target for a fix. The anchor.kind determines the nature of the fix:

    • fixture: {verb, phase, filePath}. A hook (e.g., beforeAll, afterEach) failed and cascaded. Fix the hook.
    • selector: {verb, selector}. Multiple tests share a failing Playwright locator (UI drift). Fix the selector or the element.
    • frame: {verb, frame}. Multiple tests crash at the same file:line of application code. The frame is the literal fix location.
    • signature: {verb, signature}. Tests share a normalized error signature but no specific fixture/selector/frame can be extracted (e.g., deep-stack timeouts). Look for common causes like shared timeout budgets or flaky dependencies.
    • unmatched: {testId, fileId, project}. No shared mechanism found; treat as a single-test failure.
    cluster.anchor = { kind: 'fixture', verb: 'failed', phase: 'beforeEach', filePath: 'path/to/test.spec.ts' }

  11. Understand Failure Clustering and Anchors

    main

    The dashboard uses Failure Clustering to group failures so developers can identify high-impact fixes (e.g., "one fix unblocks 12 tests"). Each failed test_run is assigned to exactly one cluster based on its Anchor.

    An anchor is determined by a priority hierarchy. When an anchor is identified, it is paired with a Playwright verb (e.g., toBeVisible, click) to ensure that different types of failures at the same location are treated as distinct clusters.

    Anchor Types and Priority

    KindAnchor (what to fix)When it fires
    fixtureA beforeAll / beforeEach / afterAll / afterEach hook in a specific fileThe failure message identifies a hook phase. This is the highest priority.
    selectorA normalized Playwright locator (stripped of UUID-shaped IDs and positional refinements like .first())The failure message contains an extractable locator.
    frameA specific file:line of app codeAn app-code frame is extracted from the stack trace or codeframe.
    unmatchedThe test itselfNo mechanism could be extracted.

    Cluster Card Organization

    The UI organizes clusters into two sections:

    1. Actionable clusters (fixture, selector, or frame): Sorted by impact (failureCount × testCount).
    2. Unmatched failures: Individual cards for failures that couldn't be clustered, showing the test title and symptom.

  12. Create Schedule Rules for cron digests

    main

    Schedule rules send periodic summaries (digests) based on a time window. These rules fan out by project: the scheduler identifies active projects in the window and sends one digest per project.

    Rule Configuration:

    • Cadence: Set to daily (HH:MM), weekly (HH:MM, Monday), or a raw cron expression.
    • Window: Defines the lookback period:
      • last_24h, last_7d, last_14d.
      • since_last_send: Looks at everything since the last successful delivery of this rule for this project (defaults to 24h, capped at 14 days).
    • Condition: always, all_clean, or no_hard_failures.
    • Project filter: Apply to all, a specific project name, or a regex pattern.

    If no projects had activity during the window, the rule is skipped with skipReason=no_activity.