Kubeflow Katib Documentation

Katib supports various statistical distributions for hyperparameters via the distribution field in the FeasibleSpace configuration. This allows for more precise optimization than simple uniform distributions, such as using logUniform for learning rates or normal for parameters centered around a specific value.

Supported Distributions

FeasibleSpace Configuration Fields

When defining a FeasibleSpace, use the following fields:

• max: Maximum value (for continuous/integer types).
• min: Minimum value (for continuous/integer types).
• list: A list of values (for Discrete or Categorical types).
• step: Defines the quantization step (e.g., q for quantized distributions).
• distribution: The type of distribution to apply (e.g., uniform, logUniform, normal, logNormal).

The KEP-2339 API automates hyperparameter optimization (HPO) for Large Language Models (LLMs) by integrating with the Training Python SDK. The workflow follows these core principles:

1. Model and Dataset Management: It uses the storage_initializer from the Training Python SDK to download pretrained models and datasets (from sources like HuggingFace or S3) into a shared PersistentVolumeClaim (PVC). This prevents redundant downloads across different trials.
2. Hyperparameter Configuration: Users define tunable hyperparameters within trainer_parameters.training_parameters and trainer_parameters.lora_config. The API identifies these tunable parameters and injects them as args into the worker container specification.
3. Orchestration: The API creates a Katib Experiment that defines the search space and objective. This experiment generates Trials, where each trial is implemented as a Kubernetes PyTorchJob. The trialTemplate within the experiment specifies the container configurations for master and worker nodes to enable parallel execution.
4. Feedback Loop: Trial results are fed back to the Experiment, which evaluates them to find the optimal hyperparameter configuration.

The Best Genotype represents the optimal cell structure for each neural network layer generated by the DARTS algorithm. It is structured into normal cells, reduce cells, and their respective concatenation ranges.

• normal: Cells located in the majority of the neural network layers.
• reduce: Cells located at the 1/3 and 2/3 marks of the total layers, representing reduction cells where operations adjacent to input nodes have a stride of two.
• normal_concat / reduce_concat: Define the range of intermediate nodes to be concatenated.

Node Structure: Each element in the normal or reduce arrays represents a node with two edges. The format is (operation, connection_index).

• connection_index 0 refers to the $C_{k-2}$ node.
• connection_index 1 refers to the $C_{k-1}$ node.

Example: [('max_pooling_3x3',0),('max_pooling_3x3',1)] means the $C_{k-2}$ node connects to the first node via max_pooling_3x3, and the $C_{k-1}$ node also connects to the first node via max_pooling_3x3.

Genotype(
  normal=[
      [('max_pooling_3x3',0),('max_pooling_3x3',1)],
      [('max_pooling_3x3',0),('max_pooling_3x3',1)],
      [('max_pooling_3x3',0),('dilated_convolution_3x3',3)],
      [('max_pooling_3x3',0),('max_pooling_3x3',1)]
    ],
    normal_concat=range(2,6),
    reduce=[
      [('dilated_convolution_5x5',1),('separable_convolution_3x3',0)],
      [('max_pooling_3x3',2),('dilated_convolution_5x5',1)],
      [('dilated_convolution_5x5',3),('dilated_convolution_5x5',2)],
      [('dilated_convolution_5x5',3),('dilated_convolution_5x5',4)]
    ],
    reduce_concat=range(2,6)
)