DOCSEARCHbyAlgolia
Submit Docs

CanaryChecker Documentation

website·Indexed 31 days ago

https://canarychecker.io

A Kubernetes-native health check platform for monitoring services via HTTP, SQL, Mongo, Redis, LDAP, and more. It supports distributed canaries, custom metrics exported to Prometheus, and flexible scripting using Go Text Templates, Common Expression Language (CEL), and JavaScript.

Tokens
44.4K
Snippets
231
Records
305
Agent score
59%

What's inside CanaryChecker

  1. Overview of Canary Checker health check types

    Canary Checker is a Kubernetes-native health check platform that supports four primary types of tests to verify service and infrastructure health:

    1. Synthetic: Periodically generates requests using protocols like HTTP, SQL, Mongo, Redis, and LDAP to test services directly.
    2. Passive: Consolidates alerts from external monitoring systems such as Prometheus, Datadog, Dynatrace, CloudWatch, and ElasticSearch.
    3. Infrastructure: Provisions infrastructure (e.g., EC2 Instances, Kubernetes Pods) and runs synthetic checks against them to verify provisioning capabilities.
    4. Integration: Runs automated test suites using tools like Playwright, JUnit, Newman, and K6 to validate end-to-end functionality.

  2. Overview of Canary Checker Health Check Platform

    Canary Checker is a Kubernetes-native health check platform that provides a single pane of glass for monitoring stack health. It supports two primary types of checks:

    1. Synthetic (Active) Checks: Generates synthetic traffic, runs queries against datasources, or executes full integration test suites to verify service and infrastructure health.
    2. Passive Checks: Consolidates alerts from monitoring stacks and non-traditional sources such as SQL, NoSQL, and CI/CD pipelines.

    Key features include "Health Checks as Code" using Kubernetes Custom Resource Definitions (CRDs), which allows integration with GitOps tools like Flux, ArgoCD, and Helm.

  3. Use Common Expression Language (CEL) in CanaryChecker expressions

    CanaryChecker uses the Common Expression Language (CEL) for expr expressions. CEL supports various types including 64-bit signed/unsigned integers (int, uint), doubles, booleans, strings, bytes, lists, maps, and null types.
    apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: currency-converter-display-cel
spec:
  http:
    - name: USD
      url: https://api.frankfurter.app/latest?from=USD&to=GBP,EUR,ILS,ZAR
      display:
        expr: "'$1 = €' + string(json.rates.EUR) + ', £' + string(json.rates.GBP) + ', ₪' + string(json.rates.ILS)"

  4. Extend canary-checker functionality using expressions

    canary-checker supports three primary ways to extend its functionality using expressions:

    1. Display Formatting: Customize the information or error messages displayed for a health check.
    2. Health Evaluation: Define the logic to determine if a check is passing or failing based on the result.
    3. Transforms: Perform arbitrary transformations on check results, such as splitting a single JSON response into multiple checks, modifying results (adding labels or changing severity), changing the namespace target, or persisting parts of a result (e.g., last log age) for use in subsequent checks.

  5. Use Go Text Templates in CanaryChecker

    CanaryChecker uses the Go Text Template library supplemented by the gomplate library for template expressions. This allows for dynamic data injection, such as printing exchange rates from an HTTP API response.
    apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: currency-converter-display-gotemplate
spec:
  http:
    - name: USD
      url: https://api.frankfurter.app/latest?from=USD&to=GBP,EUR,ILS,ZAR
      display:
        template: "$1 = €{{.json.rates.EUR}}, £{{.json.rates.GBP}}, ₪{{.json.rates.ILS}}"

  6. Verify S3 compatible endpoints using s3-protocol

    The s3-protocol check verifies that S3-compatible endpoints (such as Minio or EMC ECS) are functioning correctly. It performs three primary actions to validate permissions: lists objects (Read), puts an object into the bucket (Write), and downloads the uploaded object (Get).
    apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: s3-protocol-check
spec:
  schedule: "@every 5m"
  s3:
    - name: s3-check
      bucketName: flanksource-public
      objectPath: dummy
      region: us-east-1
      accessKey:
        valueFrom:
          secretKeyRef:
            name: aws-credentials
            key: AWS_ACCESS_KEY_ID
      secretKey:
        valueFrom:
          secretKeyRef:
            name: aws-credentials
            key: AWS_SECRET_ACCESS_KEY

  7. Extend canary-checker functionality using expressions

    canary-checker provides three primary ways to extend its functionality using expressions:

    1. Display Formatting: Customize the information or error messages displayed for a health check.
    2. Health Evaluation: Define the logic to determine if a check is passing or failing based on the result.
    3. Transforms: Perform arbitrary transformations on check results, such as:
      • Splitting a single check into multiple checks (e.g., when results are bundled in one JSON).
      • Modifying results (e.g., adding labels or changing severity).
      • Changing the namespace target of a check.
      • Saving specific parts of a result (e.g., last log age) for use in subsequent checks.

  8. Monitor folder contents with the folder check

    The folder check verifies the contents of a local or remote folder based on file size, age, and count. It is commonly used to verify backup uploads, ensure logs are being cleaned up, or monitor batch process queues for error files (e.g., .err or .log).

    Supported paths include local filesystem paths and remote protocols: smb://, sftp://, s3://, and gcs://.

  9. Run Playwright tests using the JUnit check type

    The JUnit check type in Canary Checker runs a Kubernetes pod with a specified image to execute a test suite. To integrate Playwright, configure the Playwright reporter to export JUnit XML results to the directory specified in the testResults field of the Canary specification. Canary Checker then ingests these results for health evaluation and dashboard display.
    apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: playwright-junit
spec:
  interval: 120
  severity: high
  junit:
    - testResults: "/tmp/"
      name: playwright-junit
      display:
        template: |
          ✅ {{.results.passed}} ❌ {{.results.failed}} in 🕑 {{.results.duration}}
          {{  range $r := .results.suites}}
          {{- if gt (conv.ToInt $r.failed)  0 }}
            {{$r.name}} ✅ {{$r.passed}} ❌ {{$r.failed}} in 🕑 {{$r.duration}}
          {{- end }}
          {{- end }}
      spec:
        containers:
          - name: playwright
            image: ghcr.io/flanksource/canary-playwright:latest
            command: ["/start.sh"]

  10. Verify pod reachability via Ingress using Pod check

    The Pod check is used to validate network connectivity by creating a new pod and verifying that it can be successfully reached through an Ingress controller.

  11. Use wildcards and negation in Resource Selectors

    The name, id, types, statuses, and health fields support advanced matching patterns:

    • Prefix matching: Use * at the end (e.g., name=prod-*).
    • Suffix matching: Use * at the start (e.g., name=*-backend).
    • Negation: Use ! to exclude values (e.g., health=!unhealthy).
    • Multiple values: Use commas to match any of the specified values (e.g., types=Kubernetes::Pod,Kubernetes::Deployment).

  12. Create and deploy a Canary check

    Define a Canary resource to perform scheduled HTTP checks. Use the schedule field to define the frequency (e.g., @every 30s) and the http list to specify the URLs to monitor.
    # canary.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: http-check
spec:
  schedule: "@every 30s"
  http:
    - name: basic-check
      url: https://httpbin.flanksource.com/status/200
    - name: failing-check
      url: https://httpbin.flanksource.com/status/500

    Apply to cluster

    kubectl apply -f canary.yaml