ConfigSpace

ConfigSpace provides three ways to define hyperparameters, ranging from rapid prototyping to full library development customization:

1. Inferred Hyperparameters: Define hyperparameters directly within the ConfigurationSpace constructor using Python primitives (tuples or lists). Best for rapid prototyping or simple search spaces.
2. Simple Hyperparameters: Use convenience functions like Integer(), Float(), and Categorical(). These are parameterized versions of the underlying types and are best for most use cases where you need to pass a search space to another library.
3. Direct Hyperparameters: Use the explicit hyperparameter classes (e.g., UniformIntegerHyperparameter). These offer the highest level of customization and are intended for library developers building on top of ConfigSpace.

Note: Integer, Float, and Categorical are convenience functions, not types. To perform isinstance checks, use the direct hyperparameter classes.

from ConfigSpace import ConfigurationSpace, Integer, Categorical, Float, UniformIntegerHyperparameter

# 1. Inferred (Prototyping)
cs_inferred = ConfigurationSpace({
    "a": (0, 10),           # Integer
    "b": ["cat", "dog"], # Categorical
    "c": (0.0, 1.0),       # Float
})

# 2. Simple (Most use cases)
cs_simple = ConfigurationSpace({
    "a": Integer("a", (0, 10), log=False),
    "b": Categorical("b", ["cat", "dog"], ordered=True),
    "c": Float("c", (1e-5, 1e2), log=True),
})

# 3. Direct (Library development)
cs_direct = ConfigurationSpace({
    "a": UniformIntegerHyperparameter("a", lower=0, upper=10, log=False),
    "b": CategoricalHyperparameter("b", choices=["cat", "dog"], default_value="dog"),
})

All hyperparameters in ConfigSpace inherit from the Hyperparameter base class and consist of two primary components:

1. Vectorized space: A numeric range (typically [0, 1]) and a Distribution that defines how to sample from it. Optimizers work directly with this space for efficiency.
2. Value space: The actual values returned to the user (e.g., ['cat', 'dog'] or [1, 5]).

The relationship between these spaces is managed by a Transformer. This architecture allows ConfigSpace to perform high-performance optimization on pure numeric vectors while providing meaningful domain-specific values to the user.

Key methods provided by the Hyperparameter base class include:

• sample_vector(): Samples a value from the vectorized space.
• sample_value(): Samples a vectorized value and transforms it to the value space.
• to_value(vector): Transforms a vectorized value to the value space.
• to_vector(value): Transforms a value space value to the vectorized space.
• pdf_vector() / pdf_values(): Returns the probability density function.
• legal_value() / legal_vector(): Validates if a value is within the allowed space.

ConfigSpace allows you to define forbidden clauses to prevent certain values from being sampled during hyperparameter optimization. These clauses can be used to exclude specific static values, ranges, or sets of values from a parameter's domain.

There are three main categories of forbidden clauses:

1. Static Clauses: Forbid specific values or ranges regardless of other parameters.

   • ForbiddenEqualsClause: Forbids a specific value.
   • ForbiddenInClause: Forbids a set of values.
   • ForbiddenLessThanClause / ForbiddenLessThanEqualsClause: Forbids values below a threshold.
   • ForbiddenGreaterThanClause / ForbiddenGreaterThanEqualsClause: Forbids values above a threshold.

2. Conjunctions: Combine multiple static clauses using logical operators.

   • ForbiddenAndConjunction: Forbids a configuration only if all combined clauses are met.
   • ForbiddenOrConjunction: Forbids a configuration if any of the combined clauses are met.

3. Relational Clauses: Forbid values based on the relationship between two different hyperparameters (e.g., forbidding param_a from being greater than param_b).

   • ForbiddenEqualsRelation (==)
   • ForbiddenLessThanRelation (<)
   • ForbiddenGreaterThanRelation (>)
   • ForbiddenLessThanEqualsRelation (<=)
   • ForbiddenGreaterThanEqualsRelation (>=)

Conditions allow you to create conditional hyperparameters—parameters that are only active (and sampled) if a specific condition regarding a parent hyperparameter is met. This helps limit the search space.

To implement conditions:

1. Define the hyperparameters (child and parent).
2. Add them to the ConfigurationSpace using .add().
3. Define the condition using classes like EqualsCondition, InCondition, or OrConjunction.
4. Add the conditions to the ConfigurationSpace using .add().

Warning: Avoid using EqualsCondition or InCondition on Float hyperparameters due to potential numerical rounding issues that may cause conditions to evaluate to False unexpectedly.

from ConfigSpace import ConfigurationSpace, Categorical, Float, Integer, EqualsCondition, InCondition, OrConjunction

# 1. Define hyperparameters
kernel_type = Categorical('kernel_type', ['linear', 'poly', 'rbf', 'sigmoid'])
degree = Integer('degree', bounds=(2, 4), default=2)
coef0 = Float('coef0', bounds=(0, 1), default=0.0)
gamma = Float('gamma', bounds=(1e-5, 1e2), default=1, log=True)

cs = ConfigurationSpace()
cs.add([kernel_type, degree, coef0, gamma])

# 2. Define conditions
# degree is active if kernel_type == 'poly'
cond_1 = EqualsCondition(degree, kernel_type, 'poly')

# coef0 is active if (kernel_type == 'poly' or kernel_type == 'sigmoid')
cond_2 = OrConjunction(
    EqualsCondition(coef0, kernel_type, 'poly'),
    EqualsCondition(coef0, kernel_type, 'sigmoid')
)

# gamma is active if kernel_type in ['rbf', 'poly', 'sigmoid']
cond_3 = InCondition(gamma, kernel_type, ['rbf', 'poly','sigmoid'])

# 3. Add conditions to the space
cs.add([cond_1, cond_2, cond_3])

If certain combinations of hyperparameter values are invalid, you can use Forbidden clauses to ensure they are never sampled. This is often easier than using complex conditionals when you want to explicitly exclude specific states.

Common classes include:

• ForbiddenEqualsClause: Forbids a specific value for a parameter.
• ForbiddenAndConjunction: Forbids a combination of multiple conditions (all must be met for the clause to trigger).

Add these clauses to your ConfigurationSpace using the .add() method.

from ConfigSpace import ConfigurationSpace, Categorical, Constant, ForbiddenEqualsClause, ForbiddenAndConjunction

cs = ConfigurationSpace()
penalty = Categorical("penalty", ["l1", "l2"], default="l2")
loss = Categorical("loss", ["hinge", "squared_hinge"], default="squared_hinge")
dual = Constant("dual", "False")
cs.add([penalty, loss, dual])

# Forbid: penalty is 'l1' AND loss is 'hinge'
penalty_and_loss = ForbiddenAndConjunction(
    ForbiddenEqualsClause(penalty, "l1"),
    ForbiddenEqualsClause(loss, "hinge")
)

# Forbid: dual is False AND penalty is 'l2' AND loss is 'hinge'
constant_penalty_and_loss = ForbiddenAndConjunction(
    ForbiddenEqualsClause(dual, "False"),
    ForbiddenEqualsClause(penalty, "l2"),
    ForbiddenEqualsClause(loss, "hinge")
)

cs.add([penalty_and_loss, constant_penalty_and_loss])

To add a forbidden expression (a constraint that certain combinations of hyperparameter values are disallowed) using a string, use parse_expression_from_string without specifying a conditional_hyperparameter argument.

When the conditional_hyperparameter argument is omitted, ConfigSpace interprets the resulting expression as a forbidden expression.

from ConfigSpace import ConfigurationSpace
from ConfigSpace.util import parse_expression_from_string

cs = ConfigurationSpace(
    {
        "a": (0, 10),    # Integer from 0 to 10
        "b": ["cat", "dog"],  # Categorical with choices "cat" and "dog"
        "c": (0.0, 1.0),  # Float from 0.0 to 1.0
    }
)

# Define the combination of values that are not allowed
forbidden_str = "a > 5 && c >= 0.94"

# Omitting conditional_hyperparameter makes this a forbidden expression
forbidden = parse_expression_from_string(forbidden_str, cs)

cs.add(forbidden)

You can define a search space by passing a dictionary to the ConfigurationSpace constructor. ConfigSpace infers the hyperparameter type based on the value provided:

• Integer: A tuple containing ints (e.g., (0, 10)). Uses a uniform distribution.
• Float: A tuple containing floats (e.g., (0.0, 1.0)). Uses a uniform distribution.
• Categorical: A list of elements (e.g., ["cat", "dog"]). Elements have no inherent order.

Warning on Mixed Types: If a tuple contains both int and float, the type is inferred using the first element. For example, (0, 1.0) is an integer hyperparameter, while (1.0, 10) is a float hyperparameter.

from ConfigSpace import ConfigurationSpace

cs = ConfigurationSpace(
    {
        "a": (0, 10),    # Integer from 0 to 10
        "b": ["cat", "dog"],  # Categorical with choices "cat" and "dog"
        "c": (0.0, 1.0),  # Float from 0.0 to 1.0
    }
)
print(cs)

You can incorporate prior knowledge into your search space by assigning probability distributions to hyperparameters. This is useful for black-box optimization (e.g., in SMAC) to guide the search toward likely optimal regions.

Supported distributions include:

• Normal(mean, std)
• Beta(alpha, beta)

You can also assign weights to Categorical items to represent prior probabilities.

To verify your prior, use the .pdf_values(points) method on a hyperparameter to see the probability density at specific values.

import numpy as np
from ConfigSpace import ConfigurationSpace, Float, Categorical, Beta, Normal

cs = ConfigurationSpace(
    space={
        "lr": Float(
            'lr',
            bounds=(1e-5, 1e-1),
            default=1e-3,
            log=True,
            distribution=Normal(1e-3, 1e-1)
        ),
        "dropout": Float(
            'dropout',
            bounds=(0, 0.99),
            default=0.25,
            distribution=Beta(alpha=2, beta=4)
        ),
        "activation": Categorical(
            'activation',
            items=['tanh', 'relu'],
            weights=[0.2, 0.8]
        ),
    },
    seed=1234,
)

# Verify the prior density for 'lr'
test_points = np.logspace(-5, -1, 5)
print(cs['lr'].pdf_values(test_points))