octocov

In addition to standard metrics (Code Coverage, Code to Test Ratio, and Test Execution Time), octocov accepts custom metrics via JSON files.

To use them, specify the path to the JSON file using an environment variable prefixed with OCTOCOV_CUSTOM_METRICS_ (e.g., OCTOCOV_CUSTOM_METRICS_PERF). If you have multiple files, use a separate environment variable for each.

Defining acceptable conditions

You can enforce quality gates using the acceptables array within your custom metrics JSON. This allows you to compare current values against thresholds or previous values.

Available variables in acceptables expressions:

• current.{metric_key}: The current metric value.
• prev.{metric_key}: The previous metric value (retrieved from diff.datastores:).
• diff.{metric_key}: The difference between current and previous (current.{metric_key} - prev.{metric_key}).

If a condition fails, octocov will exit with an error message indicating which condition was not met.

{
  "key": "performance_metrics",
  "name": "Performance Metrics",
  "metrics": [
    {
      "key": "response_time",
      "name": "Response Time",
      "value": 250.0,
      "unit": "ms"
    }
  ],
  "acceptables": [
    "current.response_time < 300",
    "current.response_time <= prev.response_time"
  ]
}

Central mode allows a single repository to act as a hub for collecting coverage reports from multiple other repositories. When enabled, other functions (like standard reporting) are automatically turned off.

Key Configuration Keys:

• central.root: (String) Root directory or index file for collected reports. Default is ..
• central.reports.datastores: (List) Datastores where reports are collected from. Default is local://reports.
• central.badges.datastores: (List) Datastores where badges are generated. Default is local://badges.
• central.push: (Boolean/Config) Configuration for pushing the index file and badges.
• central.reReport.datastores: (List) Additional datastores to re-store collected reports.
• central.if: (Expression) Condition for running central mode.

Usage Patterns:

• GitHub Artifacts: Use artifact:// in the sub-repos and perform badge generation via on.schedule in the central repo.
• GitHub Repositories: Use github:// in sub-repos and perform badge generation via on.push in the central repo.
• Cloud Storage (S3/GCS/BigQuery): Use the respective schemes and perform badge generation via on.schedule in the central repo.

To use octocov in your CI pipeline, follow these steps:

1. Generate coverage output: Run your tests with a coverage report output (e.g., for Go, use -coverprofile=coverage.out).
2. Initialize configuration: Run octocov init to generate a .octocov.yml file in your repository.
3. Configure Workflow: Add a step to your GitHub Actions workflow using k1LoW/octocov-action@v1.

Note: octocov can only comment on pull requests originating from the same repository due to GitHub token permissions. However, reporting to GitHub Actions Job Summaries is always permitted.

# .github/workflows/ci.yml
name: Test

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - 
        uses: actions/checkout@v3
      - 
        uses: actions/setup-go@v4
        with:
          go-version-file: go.mod
      - 
        name: Run tests with coverage report output
        run: go test ./... -coverprofile=coverage.out
      - 
        uses: k1LoW/octocov-action@v1

When running in a CI environment (CI=true), octocov can perform several automated tasks based on your .octocov.yml configuration:

• Metric Measurement: Calculates coverage, code-to-test ratio, and test execution time.
• Badge Generation: Creates visual badges for coverage, code-to-test ratio, and test execution time.
• Report Comparison (Diffing): Compares current metrics against the latest report stored in configured datastores to detect regressions.
• Pull Request Integration:
  • Comments the report on the PR.
  • Inserts the report into the PR body.
  • Adds the report to the GitHub Actions job summary.
• Data Storage: Stores reports in configured datastores (e.g., BigQuery, local files).
• Git Automation: Pushes generated files (like badges or reports) back to the repository using local Git.
• Acceptance Checks: Validates if the measured metrics meet the acceptable thresholds defined in the config.