NNI provides an automated Hyperparameter Optimization (HPO) framework to find the optimal set of hyperparameters for machine learning algorithms. Instead of using naive brute-force methods like grid search, NNI uses tuners to intelligently decide the order of hyperparameter evaluations based on historical results, significantly reducing the number of iterations required to find optimal values.
Key components of the NNI HPO workflow include:
- Tuners: Algorithms (e.g., Random Search, TPE, SMAC, PPO) that predict where the best hyperparameters are likely to be located.
- Training Platforms: Support for running experiments locally or on distributed platforms like SSH servers, Kubernetes, and AzureML.
- Web Portal: A UI to monitor training progress, visualize performance, and manage experiments.
# Example of a naive (brute-force) HPO process that NNI automates more efficiently
best_hyperparameters = None
best_accuracy = 0
for learning_rate in [0.1, 0.01, 0.001, 0.0001]:
for momentum in [i / 10 for i in range(10)]:
for activation_type in ['relu', 'tanh', 'sigmoid']:
model = build_model(activation_type)
train_model(model, learning_rate, momentum)
accuracy = evaluate_model(model)
if accuracy > best_accuracy:
best_accuracy = accuracy
best_hyperparameters = (learning_rate, momentum, activation_type)
print('Best hyperparameters:', best_hyperparameters)
