Kruize Kubernetes Resource Optimization Engine

A Metric Profile (represented as a KruizePerformanceProfile) defines how Kruize collects and interprets telemetry data to make optimization decisions.

Key Fields in a Profile:

• apiVersion & kind: Standard Kubernetes-style API identifiers.
• metadata.name: The unique identifier for the profile.
• profile_version: The version of the profile schema.
• k8s_type: The target Kubernetes distribution (e.g., openshift).
• slo: Defines the Service Level Objective, including the sloClass, the objective_function (e.g., source), and the direction (e.g., minimize).
• function_variables: An array of metrics used by the SLO. Each variable includes:
  • name: The metric identifier (e.g., cpuUsage, memoryRequest).
  • datasource: The source of the data (e.g., prometheus).
  • kubernetes_object: The scope of the metric (container or namespace).
  • aggregation_functions: A map of aggregation types (avg, min, max, sum) to their specific Prometheus query strings.

Note on Variables: Queries often use placeholders like $NAMESPACE$, $CONTAINER_NAME$, and $MEASUREMENT_DURATION_IN_MIN$ which are interpolated at runtime.

A KruizePerformanceProfile (kind: KruizePerformanceProfile) is a custom resource used to define how Kruize should calculate resource optimization recommendations. It specifies the Service Level Objectives (SLO), the objective function (e.g., minimize), and the specific metrics (function variables) to be collected from datasources like Prometheus.

Key components of a profile include:

• slo: Defines the sloClass (e.g., resource_usage), the objective_function type, and the direction (e.g., minimize).
• function_variables: A list of metrics to monitor. Each variable defines:
  • name: The metric identifier (e.g., cpuRequest, memoryUsage).
  • datasource: The source of the data (e.g., prometheus).
  • kubernetes_object: The K8s object type (e.g., container).
  • aggregation_functions: A map of aggregation types (avg, min, max, sum) to their corresponding Prometheus queries. Queries can use placeholders like $NAMESPACE$, $CONTAINER_NAME$, and $MEASUREMENT_DURATION_IN_MIN$ which are resolved at runtime.

{
  "apiVersion": "recommender.com/v1",
  "kind": "KruizePerformanceProfile",
  "metadata": {
    "name": "resource-optimization-local-monitoring"
  },
  "profile_version": 1.0,
  "k8s_type": "openshift",
  "slo": {
    "sloClass": "resource_usage",
    "objective_function": {
      "function_type": "source"
    },
    "direction": "minimize"
  },
  "function_variables": [
    {
      "name": "cpuRequest",
      "datasource": "prometheus",
      "value_type": "double",
      "kubernetes_object": "container",
      "aggregation_functions": {
        "avg": {
          "function": "avg",
          "query": "avg by(container, namespace) (kube_pod_container_resource_requests{container!='', container!='POD ', pod!='', resource='cpu', unit='core' ,namespace="$NAMESPACE$",container="$CONTAINER_NAME$"})"
        }
      }
    }
  ]
}

You can refine the optimization goals in the recommendation_settings block of your experiment request:

• Models: Choose between performance, cost, or a custom model. Custom models use model_tunables to define specific percentiles for cpu_percentile, memory_percentile, and accelerator_percentile.
• Terms: Specify the time horizon using short, medium, or long (via term_settings.terms).

Note on Modes:

• In monitor mode, multiple models and terms can be supported.
• In auto or recreate mode, only one model and one term can be set (defaults to performance and short_term).

"recommendation_settings": {
  "threshold": "0.1",
  "model_settings": {
    "models": ["balance"],
    "model_tunables": {
      "balance": {
        "memory_percentile": 89,
        "cpu_percentile": 92,
        "accelerator_percentile": 100
      }
    }
  },
  "term_settings": {
    "terms": ["short"]
  }
}

Kruize uses notification codes to communicate the status, availability of recommendations, and errors encountered during the optimization process. Each notification is categorized by a Type (INFO, NOTICE, WARNING, ERROR, CRITICAL) and a Scope, which defines the granularity of the notification.

Notification Scopes

When processing notifications, you can determine the level of granularity based on the following scope indicators:

• Recommendations Object level: Notifications related to the overall availability of recommendations.
• Timestamp level: Notifications related to specific time intervals or resource configuration settings (e.g., missing CPU/Memory requests/limits).
• Term level: Notifications related to specific metric terms or data availability (e.g., CPU/Memory records being zero or unavailable).
• Engine level: Notifications related to the internal Kruize engine or global configuration (e.g., accelerator support or pod count constraints).

Understanding these scopes allows you to filter or prioritize notifications based on whether they affect high-level recommendation availability or low-level data/engine errors.

| **Notification Id** | **Type** | **Notification Short Representation** | **Recommendations Object level** | **Timestamp level** | **Term level** | **Engine level** |
|---------------------|:--------:|:-------------------------------------:|:--------------------------------:|:-------------------:|:--------------:|:----------------:|
| 111000              |   INFO   |       RECOMMENDATIONS_AVAILABLE       |                1                 |          0          |       0         |        0         |
| 120001              |   INFO   |            NOT_ENOUGH_DATA            |                1                 |          0          |       1         |        0         |
| 128001              |   INFO   | ACCELERATOR_RECOMMENDATIONS_AVAILABLE |                0                 |          0          |       0         |        1         |
| 423001              | WARNING  |           CPU_LIMIT_NOT_SET           |                0                 |          1          |       0         |        0         |
| 523001              | CRITICAL |          CPU_REQUEST_NOT_SET          |                0                 |          1          |       0         |        0         |
| 524001              | CRITICAL |        MEMORY_REQUEST_NOT_SET         |                0                 |          1          |       0         |        0         |
| 524002              | CRITICAL |         MEMORY_LIMIT_NOT_SET          |                0                 |          1          |       0         |        0         |
| 223001              |  ERROR   |     AMOUNT_MISSING_IN_CPU_SECTION     |                0                 |          1          |       0         |        1         |
| 223002              |  ERROR   |     INVALID_AMOUNT_IN_CPU_SECTION     |                0                 |          1          |       0         |        1         |
| 224001              |  ERROR   |   AMOUNT_MISSING_IN_MEMORY_SECTION    |                0                 |          1          |       0         |        1         |
| 224002              |  ERROR   |   INVALID_AMOUNT_IN_MEMORY_SECTION    |                0                 |          1          |       0         |        1         |
| 111101              |   INFO   | SHORT_TERM_RECOMMENDATIONS_AVAILABLE  |                0                 |          1          |       0         |        0         |
| 111102              |   INFO   | MEDIUM_TERM_RECOMMENDATIONS_AVAILABLE |                0                 |          1          |       0         |        0         |
| 111103              |   INFO   |  LONG_TERM_RECOMMENDATIONS_AVAILABLE  |                0                 |          1          |       0         |        0         |
| 112101              |   INFO   |    COST_RECOMMENDATIONS_AVAILABLE     |                0                 |          0          |       1         |        0         |
| 112102              |   INFO   | PERFORMANCE_RECOMMENDATION_AVAILABLE  |                0                 |          0          |       1         |        0         |
| 112103              |   INFO   | CUSTOM_MODEL_RECOMMENDATION_AVAILABLE |                0                 |          0          |       1         |        0         |
| 112104              |   INFO   |   RUNTIMES_RECOMMENDATION_AVAILABLE   |                0                 |          0          |       1         |        0         |
| 223003              |  ERROR   |     FORMAT_MISSING_IN_CPU_SECTION     |                0                 |          1          |       1         |        0         |
| 223004              |  ERROR   |     INVALID_FORMAT_IN_CPU_SECTION     |                0                 |          1          |       1         |        0         |
| 224003              |  ERROR   |   FORMAT_MISSING_IN_MEMORY_SECTION    |                0                 |          1          |       1         |        0         |
| 224004              |  ERROR   |   INVALID_FORMAT_IN_MEMORY_SECTION    |                0                 |          1          |       1         |        0         |
| 321001              |  NOTICE  |      POD_COUNT_DERIVED_FROM_CPU       |                0                 |          1          |       1         |        0         |
| 321002              |  NOTICE  |     POD_COUNT_DERIVED_FROM_MEMORY     |                0                 |          1          |       1         |        0         |
| 323001              |  NOTICE  |         CPU_RECORDS_ARE_IDLE          |                0                 |          0          |       1         |        0         |
| 323002              |  NOTICE  |         CPU_RECORDS_ARE_ZERO          |                0                 |          0          |       1         |        0         |
| 323003              |  NOTICE  |       CPU_RECORDS_NOT_AVAILABLE       |                0                 |          0          |       1         |        0         |
| 324001              |  NOTICE  |        MEMORY_RECORDS_ARE_ZERO        |                0                 |          0          |       1         |        0         |
| 324002              |  NOTICE  |     MEMORY_RECORDS_NOT_AVAILABLE      |                0                 |          0          |       1         |        0         |
| 221003              |  ERROR   |         HOURS_CANNOT_BE_ZERO          |                0                 |          0          |       1         |        0         |
| 221004              |  ERROR   |       HOURS_CANNOT_BE_NEGATIVE        |                0                 |          0          |       1         |        0         |
| 221005              |  ERROR   |     NOT_ENOUGH_DATA_FOR_POD_COUNT     |                0                 |          0          |       1         |        0         |
| 323004              |  NOTICE  |        CPU_REQUESTS_OPTIMISED         |                0                 |          0          |       0         |        1         |
| 323005              |  NOTICE  |         CPU_LIMITS_OPTIMISED          |                0                 |          0          |       0         |        1         |
| 324003              |  NOTICE  |       MEMORY_REQUESTS_OPTIMISED       |                0                 |          0          |       0         |        1         |
| 324004              |  NOTICE  |       MEMORY_LIMITS_OPTIMISED        |                0                 |          0          |       0         |        1         |
| 328001              |  NOTICE  |       ACCELERATOR_NOT_SUPPORTED       |                0                 |          0          |       0         |        1         |
| 221001              |  ERROR   |        NUM_PODS_CANNOT_BE_ZERO          |                0                 |          0          |       0         |        1         |
| 221002              |  ERROR   |      NUM_PODS_CANNOT_BE_NEGATIVE      |                0                 |          0          |       0         |        1         |

The recommendation response follows a hierarchical structure designed to mirror Kubernetes resource specifications.

Key Data Path

For a container type experiment, the path to resource recommendations is: kubernetes_objects $\rightarrow$ containers $\rightarrow$ recommendations $\rightarrow$ data $\rightarrow$ [timestamp] $\rightarrow$ recommendation_engines $\rightarrow$ [cost|performance] $\rightarrow$ config $\rightarrow$ resources $\rightarrow$ [limits|requests].

Resource Nesting

Unlike older versions, requests and limits are nested under a resources key:

"resources": {
  "limits": {
    "memory": { "amount": 100.0, "format": "MiB" },
    "cpu": { "amount": 0.5, "format": "cores" }
  },
  "requests": {
    "memory": { "amount": 50.21, "format": "MiB" },
    "cpu": { "amount": 5.37, "format": "cores" }
  }
}

Recommendation Terms

Recommendations are often categorized by duration:

• short_term
• medium_term
• long_term

[
  {
    "cluster_name": "cluster-one-division-bell",
    "kubernetes_objects": [
      {
        "type": "deployment",
        "name": "tfb-qrh-deployment_5",
        "namespace": "default_5",
        "containers": [
          {
            "container_name": "tfb-server-1",
            "recommendations": {
              "data": {
                "2023-04-02T13:30:00.680Z": {
                  "current": {
                    "resources": {
                      "limits": { "memory": { "amount": 100.0, "format": "MiB" } }
                    }
                  },
                  "recommendation_engines": {
                    "cost": {
                      "config": {
                        "resources": {
                          "limits": { "memory": { "amount": 238.2, "format": "MiB" } }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        ]
      }
    ]
  }
]

When querying experiments in Remote Monitoring mode, the response contains detailed metadata and metric aggregation for Kubernetes objects.

Key Fields in the Response:

• experiment_id: Unique identifier for the experiment.
• mode: Set to monitor for remote monitoring.
• target_cluster: Indicates if the target is remote.
• kubernetes_objects: An array containing the target resources (e.g., type: deployment).
• containers: A map of container names to their specific metrics and images.
• results: A map of timestamps to metric data. Each entry includes:
  • metrics: Aggregated data for cpuUsage, memoryUsage, memoryRSS, cpuThrottle, memoryLimit, cpuLimit, cpuRequest, and memoryRequest.
  • aggregation_info: Contains avg, max, min, sum, and the unit format (e.g., MiB, cores).
  • interval_start_time / interval_end_time: The time window for the metrics.

[
  {
    "version": "v2.0",
    "experiment_id": "f0007796e65c999d843bebd447c2fbaa6aaf9127c614da55e333cd6bdb628a74",
    "experiment_name": "quarkus-resteasy-kruize-min-http-response-time-db_0",
    "mode": "monitor",
    "target_cluster": "remote",
    "kubernetes_objects": [
      {
        "type": "deployment",
        "name": "tfb-qrh-deployment_0",
        "containers": {
          "tfb-server-1": {
            "results": {
              "2023-04-02T13:30:00.680Z": {
                "metrics": {
                  "cpuUsage": {
                    "aggregation_info": { "avg": 0.12, "max": 0.84, "format": "cores" }
                  }
                }
              }
            }
          }
        }
      }
    ]
  }
]