kubewatch
repository·master·Indexed 21 days ago
https://github.com/robusta-dev/kubewatchA Kubernetes watcher that monitors resource changes and sends notifications to collaboration platforms including Slack, MS Teams, and Mattermost via webhooks. Includes documentation for installation via the Robusta Helm repository and detailed configuration for notification channels, resource watching logic, and deployment parameters.
What's inside kubewatch
- Kubewatch is a Kubernetes watcher that monitors resource changes in your cluster and publishes notifications to various collaboration hubs via webhooks. It supports multiple notification channels including Slack, MS Teams, Mattermost, and more. For fine-grained filtering of changes, Kubewatch can be used in conjunction with Robusta.
Configure Kubewatch via files, arguments, or environment variables
masterThe Kubewatch Controller initializes its configuration using a specific hierarchy of sources:
- Configuration File: It first attempts to read a
.kubewatch.yamlfile. - Command Line Arguments: Arguments passed during execution.
- Environment Variables: If parameters are missing from the YAML file or CLI arguments, the controller falls back to reading standard environment variables.
The resulting configuration object determines which resources are watched and which handlers are activated.
- Configuration File: It first attempts to read a
Validate required values with ValidateValue
masterThe
ValidateValuemechanism can be used to ensure specific configuration values are not empty. If a required value is missing or empty, Helm will throw an error during installation or upgrade, providing instructions on how to retrieve the missing value from a secret.Example error behavior: If a value is required but provided as an empty string via
--set:$ helm install test mychart --set path.to.value00="" 'path.to.value00' must not be empty, please add '--set path.to.value00=$PASSWORD_00' to the command. To get the current value: export PASSWORD_00=$(kubectl get secret --namespace default secretName -o jsonpath="{.data.password-00}" | base64 --decode)How Kubewatch components work together
masterKubewatch operates using three core components that manage the lifecycle of a Kubernetes event from detection to notification:
- Config: Defines the operational parameters, including which handlers to use and which filters to apply. This object is used to initialize the client.
- Controller: The engine that watches Kubernetes resources. It uses
SharedIndexInformers to listen for resource changes via the Kubernetes API Server. When events occur, the Controller places them in a rate-limiting queue, filters them based on your configuration, and then dispatches them to the appropriate Handler. - Handler: The execution layer. Once an event is filtered, the Handler is responsible for processing it and sending the notification to the configured destination (e.g., Slack, Email, etc.).
Use ExistingSecret to map sensitive data
masterIf you have existing Kubernetes secrets containing sensitive data (like passwords), you can use the
ExistingSecretschema to map those existing keys to the expected keys in your deployment.Key fields:
name: The name of the existing Kubernetes secret.keyMapping: An object that maps the expected key name in your application to the actual key name inside the existing secret.
Example usage in a deployment template:
# values.yaml name: mySecret keyMapping: password: myPasswordKey # templates/dpl.yaml (logic snippet) env: - name: PASSWORD valueFrom: secretKeyRef: name: {{ include "common.secrets.name" (dict "existingSecret" .Values.existingSecret "context" $) }} key: {{ include "common.secrets.key" (dict "existingSecret" .Values.existingSecret "key" "password") }}name: mySecret keyMapping: password: myPasswordKeyFiltering Rules for Event, Job, and Pod Resources
masterWhen
ADVANCED_FILTERS=true, Kubewatch applies specific logic to determine which events are forwarded to Robusta:Event Resources (
api/v1/Eventandevents.k8s.io/v1/Event)- Sent: Warning events that are
Created, or any event with ReasonEvicted(regardless of Type). - Filtered:
Normalevents (unless Reason isEvicted), and Warning events withUpdateorDeleteoperations.
Job Resources
- Always Sent:
CreateandDeleteevents. - Conditionally Sent (
Updateevents): Only if the Job spec changes or the Job fails (status condition containsFailed). - Filtered:
Updateevents without spec changes or failures.
Pod Resources
- Always Sent:
CreateandDeleteevents. - Conditionally Sent (
Updateevents):- Pod spec changes.
- Any container (including init containers) has
restartCount > 0. - Any container is waiting with reason
ImagePullBackOff. - The Pod is evicted.
- Any container is terminated with reason
OOMKilled.
- Filtered:
Updateevents without any of the above conditions.
All Other Resources
- All events for other resources (e.g.,
Deployments,Services,ConfigMaps) are sent without filtering.
- Sent: Warning events that are
Monitor Custom Resources (CRDs) in Kubewatch
masterTo monitor Custom Resource Definitions (CRDs), you must define the
customresourcessection in your configuration.Via Helm values file:
customresources: - group: monitoring.coreos.com version: v1 resource: prometheusrulesVia Helm
--setflag:helm install kubewatch robusta/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,resourcesToWatch.pod=true,resourcesToWatch.daemonset=true,customresources[0].group=monitoring.coreos.com,customresources[0].version=v1,customresources[0].resource=prometheusrules'Important: Custom RBAC Roles When monitoring CRDs, you must also grant Kubewatch permission to access those specific resources using
customRolesin your configuration:rbac: create: true customRoles: - apiGroups: ["monitoring.coreos.com"] resources: ["prometheusrules"] verbs: ["get", "list", "watch"]helm install kubewatch robusta/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,customRoles[0].apiGroups={monitoring.coreos.com},customRoles[0].resources={prometheusrules},customRoles[0].verbs={get,list,watch}'Add the Bitnami Common Library Chart as a dependency
masterTo use the common logic and template helpers provided by the Bitnami Common Library Chart in your own Helm chart, add it to your
Chart.yamlunder thedependenciessection. After adding the dependency, runhelm dependency updateto download the library chart.dependencies: - name: common version: 1.x.x repository: https://charts.bitnami.com/bitnami$ helm dependency updateInstall Kubewatch in a Cluster using kubectl
masterFor a quick installation using
kubectl, you can use a ConfigMap to hold your configuration and a Pod/Deployment to run the watcher.- Create the ConfigMap (ensure you update the Slack channel and token):
kubectl create -f kubewatch-configmap.yaml- Create the Pod or Deployment:
kubectl create -f kubewatch.yamlNote: A
kubewatchcontainer will be created along with akubectlsidecar container to allow communication with the API server.kubectl create -f kubewatch-configmap.yaml kubectl create -f kubewatch.yamlUninstall the Kubewatch Helm release
masterTo remove all Kubernetes components associated with the Kubewatch deployment and delete the Helm release, use the
helm deletecommand with your release name.$ helm delete my-releaseUpgrade Kubewatch Helm Chart to v3.0.0
masterUpgrading to version
3.0.0involves significant changes:- Chart labels now follow standard Helm chart practices.
- It introduces
bitnami/commonas a library chart dependency.
Important: Backwards compatibility is not guaranteed. To upgrade to
3.0.0, you should install a new release of the Kubewatch chart rather than performing an in-place upgrade. Ensure you update chart dependencies before executing the upgrade.Uninstall Kubewatch
masterTo remove the Kubewatch installation, use the standard Helm uninstall command with the name of your release.
helm uninstall <release-name>