Operator SDK

Level 5 (Auto Pilot) is the highest capability level for an Operator. It aims to eliminate manual intervention by managing the operand (the application being managed) autonomously. An Auto Pilot operator uses application-level performance indicators to make decisions regarding scaling, healing, and tuning.

Key capabilities include:

• Auto-scaling: Scaling the operand up or down based on specific operand metrics (e.g., requests per second).
• Auto-Healing: Automatically fixing unhealthy operands based on metrics, alerts, or logs, and preventing transitions into unhealthy states.
• Auto-tuning: Dynamically tuning the operand to match workload patterns, such as shifting workloads to better-suited nodes or modifying configurations.
• Abnormality detection: Identifying deviations from a standard performance profile.

Example Scenario: A database operator monitors query load and automatically scales read-only slave replicas. It detects subpar index performance and rebuilds indexes during low-load periods. It also monitors performance baselines to alert on slow queries and can automatically migrate database files to a higher-performance PersistentVolume class if high disk latency is detected.

When using an Ansible-based Operator, the Operator automatically passes variables from your Custom Resource (CR) spec to Ansible as extra-vars. This behavior is equivalent to passing variables via the --extra-vars flag in ansible-playbook.

In addition to the keys defined in your spec, the Operator injects a special ansible_operator_meta object containing the CR's name and namespace. It also provides the full CR and the CR's spec under keys prefixed with the API group (e.g., _app_example_com_database).

To access the CR metadata in your Ansible tasks, use dot notation on the ansible_operator_meta object.

# Example Custom Resource
apiVersion: "cache.example.com/v1alpha1"
kind: "Memcached"
metadata:
  name: "memcached-sample"
spec:
  message: "Hello world 2"
  newParameter: "newParam"

# --- Resulting structure passed to Ansible ---
{
  "ansible_operator_meta": {
    "name": "memcached-sample",
    "namespace": "<cr-namespace>"
  },
  "message": "Hello world 2",
  "new_parameter": "newParam",
  "_cache_example_com_memcached": { <Full CR> },
  "_cache_example_com_memcached_spec": { <Full CR .spec> }
}

# --- Accessing metadata in an Ansible task ---
- debug:
    msg: "name: {{ ansible_operator_meta.name }}, {{ ansible_operator_meta.namespace }}"

In clusters with multi-architecture compute nodes, pods may be scheduled on nodes with incompatible architectures, leading to exec format error and ImagePullBackoff events.

To prevent this, use Kubernetes nodeAffinity in your PodSpec or PodTemplateSpec to restrict pods to nodes that match the supported kubernetes.io/arch and kubernetes.io/os of your images.

It is a best practice to use requiredDuringSchedulingIgnoredDuringExecution to guarantee compatibility. You can also use preferredDuringSchedulingIgnoredDuringExecution to steer pods toward architectures where the operator performs better.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/arch
          operator: In
          values:
          - amd64
          - arm64
        - key: kubernetes.io/os
          operator: In
          values:
          - linux

Missing Resource Requests

If requests are not set:

• ResourceQuota configurations may reject Pod creation if a LimitRange is not present to provide defaults.
• The Kubernetes scheduler cannot make informed placement decisions, potentially leading to resource shortages.
• Pods are more susceptible to being OOM Killed (Memory) or experiencing CPU starvation during contention.

Missing Resource Limits

If limits are not set:

• A single container could consume all available cluster resources, affecting other workloads.
• Containers are more vulnerable to Denial of Service (DoS) attacks.

Behavior when limits are reached

• Memory Limits: If a container exceeds its memory limit, it is typically terminated with the reason OOM Killed.
• CPU Limits: CPU is a "compressible" resource. If a container hits its CPU limit, Kubernetes will throttle the container using the kernel, resulting in degraded performance rather than termination.