HPOBench: Reproducible Benchmarks for Multi-Fidelity Hyperparameter Optimization

To remove all data, downloaded containers, and caches, use the provided cleanup script:

python ./hpobench/util/clean_up_script.py --help

Manual Cleanup Locations

If you need to delete specific parts, check these directories:

• HPOBench Data:
  • $XDG_CONFIG_HOME (~/.config/hpobench)
  • $XDG_CACHE_HOME (~/.cache/hpobench)
  • $XDG_DATA_HOME (~/.local/share/hpobench)
• OpenML Data: ~/.openml/
• Singularity Containers: Use singularity cache clean.
• Stale Sockets: Check /tmp/hpobench_socket for leftover files from crashes.

To install HPOBench, clone the repository and install it via pip. It is recommended to use a virtual environment.

Note: This installation does not include Singularity (version 3.6). You must install Singularity separately following the Singularity quick start guide.

git clone https://github.com/automl/HPOBench.git
cd HPOBench 
pip install .

Benchmarks can be run locally without Singularity, but you must manually install the required dependencies, which may cause conflicts. For example, the XGBoost benchmark can be installed using an extra pip index.

pip install .[xgboost]

from hpobench.benchmarks.ml.xgboost_benchmark_old import XGBoostBenchmark

b = XGBoostBenchmark(task_id=167149)
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(configuration=config,
                                   fidelity={"n_estimators": 128, "dataset_fraction": 0.5}, rng=1)

Settings are managed via an hpobenchrc file (YAML format), which is automatically generated upon the first use of the library.

• Default Location: $XDG_CONFIG_HOME (defaults to ~/.config/hpobench).
• Unix Sockets: For running containers, sockets are located in $TEMP_DIR (defaults to /tmp).

It is recommended to inspect this file to manage where containers and datasets are stored.

Singularity 'Invalid Image format' exception

• Ensure you are using Singularity version > 3.
• If on the Freiburg Meta-Cluster, set the path: export PATH=/usr/local/kislurm/singularity-3.5/bin/:$PATH.

SystemError: Could not start an instance of the benchmark

If the error SystemError: Could not start an instance of the benchmark. Retried 5 times occurs, but the container starts fine via singularity instance start, check for corrupted instance files:

• Look in ~/.singularity/instances/sing/$HOSTNAME/*/.
• Delete any files that do not end with } and retry.

HPOBench provides containerized benchmarks to isolate dependencies and ensure reproducibility. These use Singularity. You can evaluate a configuration by calling the objective_function method. You can specify a fidelity dictionary to control multi-fidelity evaluations, or omit it to run at the highest budget.

from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark

# Initialize benchmark
b = SliceLocalizationBenchmark(rng=1)

# Get configuration space and sample a configuration
config = b.get_configuration_space(seed=1).sample_configuration()

# Evaluate with a specific fidelity
result_dict = b.objective_function(configuration=config, fidelity={"budget": 100}, rng=1)

# Evaluate without specifying fidelity (returns results on the highest budget)
result_dict = b.objective_function(configuration=config, rng=1)

For any benchmark, you can retrieve the configuration space, the fidelity space, and general meta-information using the following methods:

from hpobench.container.benchmarks.nas.tabular_benchmarks import SliceLocalizationBenchmark

b = SliceLocalizationBenchmark(task_id=167149, rng=1)
cs = b.get_configuration_space(seed=1)
fs = b.get_fidelity_space(seed=1)
meta = b.get_meta_information()