DOCSEARCHbyAlgolia
Submit Docs

Node Healthcheck Operator

repository·main·Indexed 41 days ago

https://github.com/medik8s/node-healthcheck-operator

An operator that automates the detection of unhealthy Kubernetes nodes and triggers remediation (fencing/remediation) to ensure workloads, such as StatefulSets requiring at-most-one semantics, are safely recovered. It monitors NodeConditions against criteria defined in NodeHealthCheck Custom Resources and supports safety thresholds to prevent mass outages during control plane failures.

Tokens
8.2K
Snippets
18
Records
36
Agent score
50%

What's inside Node Healthcheck Operator

  1. Understand NHC Certsuite Best Practices Exceptions

    main
    The Node HealthCheck (NHC) operator requires specific deviations from Red Hat's certsuite (cloud-native best practices) to fulfill its architectural role of cluster-wide node monitoring. These exceptions cover node tolerations for monitoring control-plane/infrastructure nodes and specific RBAC permissions for mTLS metrics authentication.

  2. Use RemediationTemplates and EscalatingRemediations

    main

    NHC uses ObjectReferences to remediation templates provided by remediation providers. You must choose one of two mutually exclusive strategies:

    1. remediationTemplate: A single template used for all remediations.
    2. escalatingRemediations: A list of templates used sequentially. Each entry includes an order (lower is earlier) and a timeout.

    Escalation Features:

    • Timeouts: When a timeout is reached, NHC adds the remediation.medik8s.io/nhc-timed-out annotation to the remediation CR so the remediator can cancel its efforts.
    • Failure Signaling: If a remediator fails, it can set a status condition of type Succeeded with status False on the remediation CR. NHC will immediately move to the next remediation in the list without waiting for the timeout.
    escalatingRemediations:
  - remediationTemplate:
      apiVersion: self-node-remediation.medik8s.io/v1alpha1
      kind: SelfNodeRemediationTemplate
      namespace: <SNR namespace>
      name: self-node-remediation-automatic-strategy-template
    order: 1
    timeout: 300s
  - remediationTemplate:
      apiVersion: reprovision.example.com/v1
      kind: ReprovisionRemediationTemplate
      namespace: example
      name: reprovision-remediation-template
    order: 2
    timeout: 30m

  3. Handling cluster upgrades and manual pausing

    main

    NHC includes mechanisms to prevent remediation from interfering with maintenance activities:

    • Automatic Upgrade Detection: On OpenShift, NHC monitors the ClusterVersionOperator. If it detects a cluster upgrade is in progress, it will stop remediating new unhealthy nodes to avoid interfering with worker reboots or node rebalancing.
    • Manual Pausing: Cluster admins can manually pause the NodeHealthCheck CR to prevent any new remediation from starting during upgrades or other sensitive operations.

  4. Configure Node Selection with LabelSelectors

    main

    Use a selector (a Kubernetes LabelSelector) to define the set of nodes the NHC will observe.

    Best Practices:

    • Worker Nodes: To avoid accidentally remediating control plane nodes in compact clusters, exclude them using DoesNotExist for both node-role.kubernetes.io/control-plane and node-role.kubernetes.io/master.
    • Control Plane Nodes: Use operator: Exists for node-role.kubernetes.io/control-plane (or master on older clusters).
    • Warning: Do not create overlapping configurations that select the same nodes, and avoid configurations that select both worker and control plane nodes simultaneously.
    # Example: Selecting worker nodes
selector:
  matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
    - key: node-role.kubernetes.io/master
      operator: DoesNotExist

  5. Enable Storm Recovery for Mass Failure Scenarios

    main

    Storm Recovery is an advanced feature that prevents the system from being overwhelmed by too many concurrent remediations during mass failure events. It is enabled by setting spec.stormCooldownDuration.

    How it works:

    1. Storm Hits: If healthyNodes < minHealthy, NHC enters Storm Recovery Active mode and blocks all new remediations.
    2. Recovery Starts: Once healthyNodes >= minHealthy, NHC enters the Regaining Health (Cooldown) state and starts the stormCooldownDuration timer.
    3. Exit: Only after the cooldown timer elapses does NHC exit storm mode and resume creating remediations for any remaining unhealthy nodes.

    When to use: Use this when your infrastructure cannot handle unlimited concurrent remediations or when you prefer a 'wait and see' approach during mass failures.

  6. How NHC generates Remediation resources from templates

    main

    When NHC detects an unhealthy node, it automatically creates a remediation CR based on a provided template. The generated CR follows these mapping rules:

    • apiVersion: Same as the template.
    • kind: The template's kind with the Template suffix removed (e.g., MyRemediationTemplate becomes MyRemediation).
    • namespace: Same as the template.
    • name: The name of the unhealthy node.
    • spec: A direct copy of the template's spec.template.spec.
    • ownerReferences: Set to the NHC CR and, if available (OKD/OpenShift), the node's machine.

    Example generated remediation for a node named unhealthy-node-name using the template above:

    apiVersion: remediator.company.io/v1
kind: MyRemediation
metadata:
  name: unhealthy-node-name
  namespace: test-namespace
  ownerReferences:
    - kind: NodeHealthCheck
      apiVersion: remediation.medik8s.io/v1alpha1
      name: nhc-snr-worker
      uid: some-uid
spec:
  strategy: reboot
  timeout: 5m

  7. How the Node Healthcheck operator works

    main

    The Node Healthcheck (NHC) operator automates failure detection and remediation for Kubernetes nodes. It works by monitoring a Node's NodeConditions against specific criteria and thresholds defined in a NodeHealthCheck Custom Resource (CR).

    When a node is deemed to be in a failed state, the NHC controller instantiates remediation Custom Resources based on templates defined in the NHC CR. These templates allow you to use various remediation methods (e.g., ClusterAPI, OpenShift MachineAPI, BMC, Watchdog, or software reboots) to fence workloads or remediate the node. Once the node recovers and becomes healthy again, NHC automatically deletes the remediation CR to signal successful recovery.

  8. Configure NHC Node Tolerations for Cluster-Wide Monitoring

    main

    To ensure the NHC operator can monitor all nodes in the cluster (including master, control-plane, and infrastructure nodes) and remain operational during node pressure, it must use specific non-default tolerations.

    Key requirements:

    • Master/Control-Plane/Infra: Must tolerate NoSchedule to allow scheduling on these nodes.
    • Infrastructure Nodes: Must tolerate NoExecute to prevent pod eviction during resource pressure, ensuring continuous monitoring.
    • Priority: The operator uses priorityClassName: system-cluster-critical to reflect its infrastructure-level importance.
    tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/infra"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/infra"
    operator: "Exists"
    effect: "NoExecute"

  9. Configure HealthyDelay for Post-Remediation Stability

    main

    The healthyDelay field defines a grace period after a node reports as healthy before NHC removes its associated remediation and potentially removes taints. This prevents 'flapping' nodes from immediately re-entering the cluster.

    Manual Intervention: If healthyDelay is set to a negative value, NHC will never automatically consider the node healthy. You can manually intervene using one of two methods:

    1. Global (CR-level): Modify the NodeHealthCheck CR by setting healthyDelay to 0s or removing the field. This affects all nodes managed by that CR.
    2. Node-specific: Add the annotation remediation.medik8s.io/manually-confirmed-healthy (with any value) to the specific Node object. NHC will ignore the delay for that node, clear the annotation once the node meets all other healthy criteria, and delete the remediation.

  10. Protecting against control plane failures and mass outages

    main

    To prevent the operator from triggering mass remediation during control plane failures or large-scale outages, the NodeHealthCheck CR supports safety thresholds. If the cluster falls below these thresholds, NHC will stop starting new remediations.

    Supported safety parameters:

    • Minimum healthy nodes: Define a minimum number of healthy nodes required to allow remediation. This can be specified as an absolute number or a percentage.
    • maxUnhealthy: Specify the maximum number of unhealthy nodes allowed before remediation is paused. This is useful for high-availability (N-1 or N-2) scenarios where the system must tolerate a specific number of failures without impacting overall functionality.

  11. Getting started with the Node Healthcheck Operator

    main

    The Node Healthcheck (NHC) operator is designed for failure detection and handling of unhealthy Kubernetes nodes. To use the operator, you should follow these primary steps:

    1. Install the operator: Follow the installation guide to deploy the operator into your cluster.
    2. Configure the operator: Use the NHC Custom Resource (CR) to define your healthcheck logic and node handling policies.
    3. Understand the workflow: Review the controller workflow to understand how the operator detects failures and executes actions.
    4. Troubleshoot: If you encounter issues, refer to the troubleshooting guidelines.

    If you cannot find the information you need, you can reach out via the Google group.

  12. Manual Setup for Platform Prometheus Metrics

    main

    If you are deploying the operator manually (without OLM), you must ensure the operator's namespace is labeled so that OpenShift's cluster monitoring can discover it. Use the openshift.io/cluster-monitoring: "true" label on the namespace where the operator is deployed (e.g., openshift-workload-availability).

    kubectl label namespace openshift-workload-availability openshift.io/cluster-monitoring=true --overwrite