OpenKruise Documentation

OpenKruise is a CNCF incubating project that extends Kubernetes core controllers for advanced workload and application management. Its key feature sets include:

• Advanced Workloads: Enhanced management for stateless, stateful, daemon, and job workloads, supporting in-place updates, configurable scale/upgrade strategies, and parallel operations. Key resources include CloneSet, Advanced StatefulSet, Advanced DaemonSet, BroadcastJob, and AdvancedCronJob.
• Sidecar Container Management: Simplifies sidecar injection, in-place updates, and startup/termination control via SidecarSet, Container Launch Priority, and Sidecar Job Terminator.
• Multi-domain Management: Manages applications across different node pools, zones, architectures (x86/arm), or node types using WorkloadSpread and UnitedDeployment.
• Enhanced Operations: Tools for container recreation (ContainerRecreateRequest), image pre-downloading (ImagePullJob), resource distribution (ResourceDistribution), persistent pod states (PersistentPodState), and custom probes (PodProbeMarker).
• Application Protection: Protects resources from cascading deletion and manages application availability via PodUnavailableBudget.

OpenKruise is a CNCF incubating project that provides extended workload and application management capabilities beyond the standard Kubernetes core controllers. Its core features include:

• Advanced Workloads: Manages stateless, stateful, daemon, and job-type applications with features like in-place upgrades, configurable scaling/release strategies, and concurrency control. Key types include CloneSet, Advanced StatefulSet, Advanced DaemonSet, BroadcastJob, and AdvancedCronJob.
• Sidecar Container Management: Simplifies sidecar injection and in-place upgrades via SidecarSet. It also provides Container Launch Priority for startup order control and Sidecar Job Terminator for managing sidecar lifecycles in Jobs.
• Multi-zone Management: Facilitates deployment across different node resource pools, availability zones, or architectures (x86/arm) using WorkloadSpread or UnitedDeployment.
• Enhanced Operations: Includes in-place container restarts, image pulling on specific nodes, cross-namespace resource distribution (ResourceDistribution), persistent pod states (PersistentPodState), and custom probe capabilities (PodProbeMarker).
• Application Security: Provides deletion protection for Kubernetes resources/pods and PodUnavailableBudget for enhanced disruption protection.

The Enhanced LivenessProbe solution replaces the native kubelet probe logic with a distributed system to provide better application protection and scalability:

1. EnhancedLivenessProbeWebhook (Kruise-manager): Converts native livenessProbe configs into a JSON string stored in the apps.kruise.io/container-probe-context pod annotation.
2. LivenessNodeProbeController (Kruise-manager): Converts the annotation data into a NodePodProbe Custom Resource (CR).
3. LivenessProbeDetector (Kruise-daemon): A component in the Kruise-daemon that executes the actual probes (TCP, HTTP, EXEC, or RPC) based on the NodePodProbe resource and reports results back to the Pod annotations.
4. EnhancedLivenessProbeController (Kruise-manager): Analyzes probe failures and decides whether to restart containers. It uses a ContainerRecreateRequest (CRR) to trigger restarts, ensuring high availability by respecting PodUnavailableBudget (maxUnavailable protection) and applying resilience algorithms (token bucket, blacklists, etc.).

The ConfigMapSet resource allows you to manage multiple versions of configuration files (Env/files) alongside workloads like CloneSet. This enables decoupling image releases from configuration releases, which is useful for large-scale data/model loading scenarios where you want to preserve validation progress or serve different data versions without changing the application image.

Key capabilities include:

• Version management via a RevisionManager ConfigMap (RMC).
• Container selection (static or dynamic).
• Reload sidecar container injection.
• Flexible update strategies (ReStart, PostHook, or HotUpdate).
• Rollback support.

apiVersion: apps.kruise.io/v1alpha1
kind: ConfigMapSet
metadata:
  name: deploy-cms
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: sample
  data: 
    application.yaml: |
      server:
        port: 6666
  containers:
  - name: main
    mountPath: /data/conf1
  reloadSidecarConfig:
    type: k8s
    config:
      name: reload-sidecar
      image: openkruise/reload-sidecar:v1.0.0
      restartPolicy: Always
  effectPolicy:
    type: HotUpdate

CloneSet scale-down logic includes rankers to ensure pods are deleted in a way that respects topology constraints and minimizes node stacking.

When scaling down, the controller uses two primary ranking mechanisms:

1. SameNodeRanker: Prioritizes deleting pods that are stacked on the same node. Pods that are the sole occupant of a node are assigned a rank of 0 and are preserved longer. For nodes with multiple pods, pods are ranked based on their index after sorting.
2. SpreadConstraintsRanker: Uses Pod Topology Spread Constraints defined in the CloneSet template to distribute pod deletion across different topology domains (e.g., zones or regions).

If topologySpreadConstraints are defined in the cloneset.spec.template.spec, the controller uses SpreadConstraintsRanker to rank pods across topology values, but still applies SameNodeRanker to sort pods within the same node/topology to ensure efficient node evacuation.