Testomatio Reporter

The rid (Report ID) parameter allows you to report the same test multiple times within a single run. This is useful when running the same test across different environments (e.g., Windows vs Linux, or different browsers) where you want each execution to be recorded separately even if they share the same test_id.

By providing a unique rid for each environment, Testomat.io recognizes them as distinct executions of the same test case.

{
  "tests": [
    {
      "title": "Should login successfully",
      "status": "passed",
      "test_id": "@Ta0b0c0d0",
      "rid": "chrome-login:windows",
      "run_time": 0.5
    },
    {
      "title": "Should login successfully",
      "status": "failed",
      "test_id": "@Ta0b0c0d0",
      "rid": "chrome-login:linux",
      "run_time": 0.6
    }
  ]
}

The reporter automatically deduplicates retried tests. If a single test is run multiple times, the results are combined into one Testomat.io test entry based on the Allure historyId.

• Status: The final status reflects the last attempt. If the final attempt passes but earlier attempts failed, the test is marked as failed with combined failure information.
• Failure Info: If any retry attempt fails, all failure messages and stack traces from all failed attempts are combined into the report.

The reporter automatically detects and uploads files as artifacts if they are referenced in the test output using the file:// URL pattern. This works across different platforms and supports the following formats:

• Unix/Linux (absolute): file:///absolute/path/to/file.png
• Unix/Linux (relative): file://relative/path/to/file.png
• Windows (backslashes): file:/C:\Users\username\path\to\file.png
• Windows (forward slashes): file://C:/Users/username/path/to/file.png

In NUnit, if the XML report contains a <ResultFiles> section (common in Playwright DotNet), all items within it will also be automatically uploaded.

The debug file uses JSONL (JSON Lines) format. Unlike a standard JSON array, each line in the file is a standalone, valid JSON object. This allows for streaming processing and incremental writing of test data.

Entry Structure

Every entry must include a time field t representing the time elapsed since the previous action.

{
  "t": "+150ms"
  // ... action-specific fields
}

Time Field (t) Format

• Format: "+{milliseconds}ms" (e.g., "+150ms", "+2.5s")
• Represents the duration passed since the previous entry.

{"t":"+0ms","datetime":"2024-01-15T10:30:00.000Z","timestamp":1705315800000}
{"t":"+5ms","data":"variables","testomatioEnvVars":{"TESTOMATIO":"abc123"}}
{"t":"+10ms","action":"createRun","params":{"title":"Test Run"}}

By default, when a beforeEach hook fails in WebdriverIO, the execution stops and remaining tests are skipped. These skipped tests are typically NOT reported to Testomatio, leading to incomplete reports.

This package solves this by:

1. Tracking beforeEach hook failures per suite.
2. Parsing spec files using AST (Abstract Syntax Tree) to identify all test titles.
3. Reporting all tests (including those that would normally be skipped) as failed when a beforeEach hook failure is detected. This includes support for parametrized tests.

You can link test results to existing tests in Testomat.io using the tid:// pattern in your test output. Use the 8-character test ID from Testomat.io (excluding the @T prefix from the URL).

Pattern: tid://@T{test_id}

Methods of linking:

1. Test Output: Print tid://@T8acca9eb to stdout/console during test execution.
2. Test Name: Add the ID to the test name (e.g., test name @T8acca9eb).
3. Suite ID: Use @S{suite_id} to place a test inside a specific suite (e.g., test name @S1428a8fa).
4. Source Code Comments: For supported languages, add the ID as a comment in the test body (e.g., // @T8acca9eb).

<testcase name="User Registration Test" classname="UserTests">
  <system-out><![CDATA[
Starting user registration test
tid://@T8acca9eb
User registration completed successfully
  ]]></system-out>
</testcase>

Testomat.io supports two privacy modes for artifacts:

1. Public Access (Default): Artifacts are uploaded with public-read permissions. Files are accessible via a permanent public URL. This is ideal for displaying images directly in Testomat.io, GitHub, or GitLab Pipes.

2. Private Access: Artifacts are stored securely. To use this mode:

   • Set the environment variable TESTOMATIO_PRIVATE_ARTIFACTS=1.
   • Provide the S3 credentials in the Testomat.io Application under Settings > Artifacts so the application can generate pre-signed links.
   • Links generated in this mode are pre-signed and expire automatically after 10 minutes.

When using CI artifact viewers (like Jenkins) that cannot access file:// links, you need to ensure your HTML report is 'portable'.

By default, if S3 uploading is not configured, the pipe copies artifacts (like screenshots) to an artifacts/ folder next to the HTML report and uses relative links (e.g., ./artifacts/failure.png).

To ensure portability even when S3 artifact uploading is configured, set: TESTOMATIO_HTML_REPORT_COPY_ARTIFACTS=1.

Note: When using portable links, you must archive or publish the entire report folder (e.g., html-report/**), not just the single .html file, so that the artifacts/ directory is included.