kubewatch

repository·master·Indexed 21 days ago

https://github.com/robusta-dev/kubewatch

A Kubernetes watcher that monitors resource changes and sends notifications to collaboration platforms including Slack, MS Teams, and Mattermost via webhooks. Includes documentation for installation via the Robusta Helm repository and detailed configuration for notification channels, resource watching logic, and deployment parameters.

Tokens
12.2K
Snippets
59
Records
76
Agent score
72%

What's inside kubewatch

  1. Overview of Kubewatch

    master
    Kubewatch is a Kubernetes watcher that monitors resource changes in your cluster and publishes notifications to various collaboration hubs via webhooks. It supports multiple notification channels including Slack, MS Teams, Mattermost, and more. For fine-grained filtering of changes, Kubewatch can be used in conjunction with Robusta.
  2. Configure Kubewatch via files, arguments, or environment variables

    master

    The Kubewatch Controller initializes its configuration using a specific hierarchy of sources:

    1. Configuration File: It first attempts to read a .kubewatch.yaml file.
    2. Command Line Arguments: Arguments passed during execution.
    3. Environment Variables: If parameters are missing from the YAML file or CLI arguments, the controller falls back to reading standard environment variables.

    The resulting configuration object determines which resources are watched and which handlers are activated.

  3. Validate required values with ValidateValue

    master

    The ValidateValue mechanism can be used to ensure specific configuration values are not empty. If a required value is missing or empty, Helm will throw an error during installation or upgrade, providing instructions on how to retrieve the missing value from a secret.

    Example error behavior: If a value is required but provided as an empty string via --set:

    $ helm install test mychart --set path.to.value00=""
        'path.to.value00' must not be empty, please add '--set path.to.value00=$PASSWORD_00' to the command. To get the current value:
    
            export PASSWORD_00=$(kubectl get secret --namespace default secretName -o jsonpath="{.data.password-00}" | base64 --decode)
  4. How Kubewatch components work together

    master

    Kubewatch operates using three core components that manage the lifecycle of a Kubernetes event from detection to notification:

    1. Config: Defines the operational parameters, including which handlers to use and which filters to apply. This object is used to initialize the client.
    2. Controller: The engine that watches Kubernetes resources. It uses SharedIndexInformers to listen for resource changes via the Kubernetes API Server. When events occur, the Controller places them in a rate-limiting queue, filters them based on your configuration, and then dispatches them to the appropriate Handler.
    3. Handler: The execution layer. Once an event is filtered, the Handler is responsible for processing it and sending the notification to the configured destination (e.g., Slack, Email, etc.).
  5. Use ExistingSecret to map sensitive data

    master

    If you have existing Kubernetes secrets containing sensitive data (like passwords), you can use the ExistingSecret schema to map those existing keys to the expected keys in your deployment.

    Key fields:

    • name: The name of the existing Kubernetes secret.
    • keyMapping: An object that maps the expected key name in your application to the actual key name inside the existing secret.

    Example usage in a deployment template:

    # values.yaml
    name: mySecret
    keyMapping:
      password: myPasswordKey
    
    # templates/dpl.yaml (logic snippet)
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "common.secrets.name" (dict "existingSecret" .Values.existingSecret "context" $) }}
                  key: {{ include "common.secrets.key" (dict "existingSecret" .Values.existingSecret "key" "password") }}
    name: mySecret
    keyMapping:
      password: myPasswordKey
  6. Filtering Rules for Event, Job, and Pod Resources

    master

    When ADVANCED_FILTERS=true, Kubewatch applies specific logic to determine which events are forwarded to Robusta:

    Event Resources (api/v1/Event and events.k8s.io/v1/Event)

    • Sent: Warning events that are Created, or any event with Reason Evicted (regardless of Type).
    • Filtered: Normal events (unless Reason is Evicted), and Warning events with Update or Delete operations.

    Job Resources

    • Always Sent: Create and Delete events.
    • Conditionally Sent (Update events): Only if the Job spec changes or the Job fails (status condition contains Failed).
    • Filtered: Update events without spec changes or failures.

    Pod Resources

    • Always Sent: Create and Delete events.
    • Conditionally Sent (Update events):
      • Pod spec changes.
      • Any container (including init containers) has restartCount > 0.
      • Any container is waiting with reason ImagePullBackOff.
      • The Pod is evicted.
      • Any container is terminated with reason OOMKilled.
    • Filtered: Update events without any of the above conditions.

    All Other Resources

    • All events for other resources (e.g., Deployments, Services, ConfigMaps) are sent without filtering.
  7. Monitor Custom Resources (CRDs) in Kubewatch

    master

    To monitor Custom Resource Definitions (CRDs), you must define the customresources section in your configuration.

    Via Helm values file:

    customresources:
      - group: monitoring.coreos.com
        version: v1
        resource: prometheusrules

    Via Helm --set flag:

    helm install kubewatch robusta/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,resourcesToWatch.pod=true,resourcesToWatch.daemonset=true,customresources[0].group=monitoring.coreos.com,customresources[0].version=v1,customresources[0].resource=prometheusrules'

    Important: Custom RBAC Roles When monitoring CRDs, you must also grant Kubewatch permission to access those specific resources using customRoles in your configuration:

    rbac:
      create: true 
      customRoles:
        - apiGroups: ["monitoring.coreos.com"]
          resources: ["prometheusrules"]
          verbs: ["get", "list", "watch"]
    helm install kubewatch robusta/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,customRoles[0].apiGroups={monitoring.coreos.com},customRoles[0].resources={prometheusrules},customRoles[0].verbs={get,list,watch}'
  8. Add the Bitnami Common Library Chart as a dependency

    master

    To use the common logic and template helpers provided by the Bitnami Common Library Chart in your own Helm chart, add it to your Chart.yaml under the dependencies section. After adding the dependency, run helm dependency update to download the library chart.

    dependencies:
      - name: common
        version: 1.x.x
        repository: https://charts.bitnami.com/bitnami
    $ helm dependency update
  9. Install Kubewatch in a Cluster using kubectl

    master

    For a quick installation using kubectl, you can use a ConfigMap to hold your configuration and a Pod/Deployment to run the watcher.

    1. Create the ConfigMap (ensure you update the Slack channel and token):
    kubectl create -f kubewatch-configmap.yaml
    1. Create the Pod or Deployment:
    kubectl create -f kubewatch.yaml

    Note: A kubewatch container will be created along with a kubectl sidecar container to allow communication with the API server.

    kubectl create -f kubewatch-configmap.yaml
    kubectl create -f kubewatch.yaml
  10. Upgrade Kubewatch Helm Chart to v3.0.0

    master

    Upgrading to version 3.0.0 involves significant changes:

    • Chart labels now follow standard Helm chart practices.
    • It introduces bitnami/common as a library chart dependency.

    Important: Backwards compatibility is not guaranteed. To upgrade to 3.0.0, you should install a new release of the Kubewatch chart rather than performing an in-place upgrade. Ensure you update chart dependencies before executing the upgrade.