gplearn Documentation

gplearn is an extension of the scikit-learn library designed to perform Genetic Programming (GP) via symbolic regression.

Symbolic regression is a machine learning technique that identifies the underlying mathematical expression describing the relationship between independent variables and dependent variable targets. It works by:

1. Building a population of random formulas.
2. Evolving these formulas over successive generations.
3. Selecting the 'fittest' individuals to undergo genetic operations (breeding, mutation, etc.) to move closer to the target function.

Because GP is a stochastic optimization process, you can control the randomness of the evolution using the random_state parameter in the estimator.

gplearn provides three main types of estimators designed for different machine learning tasks:

• SymbolicRegressor: Used for regression problems to identify underlying mathematical expressions.
• SymbolicClassifier: Used for binary classification tasks.
• SymbolicTransformer: Used for automated feature engineering. While designed for regression, it is also suitable for binary classification.

gplearn is a Python library for Genetic Programming (GP) specifically focused on solving symbolic regression problems. It aims to identify mathematical expressions that describe the relationship between independent variables and dependent variable targets by evolving a population of formulas through successive generations.

The library is designed to be compatible with the scikit-learn ecosystem, following its API patterns and integrating seamlessly with scikit-learn pipelines and grid search modules.

The SymbolicTransformer performs automated feature engineering by finding non-linear interactions that can be used by a second estimator. Unlike the regressor which minimizes error, the transformer maximizes the correlation between the predicted value and the target.

Correlation Methods

• Pearson correlation (default): Best if the transformed variables will be fed into a linear model.
• Spearman rank-order correlation: Best if the next estimator is tree-based (e.g., Random Forest or Gradient Boosting Machine).

Selection Process

1. The transformer evaluates the best programs from the final generation, controlled by the hall_of_fame parameter.
2. It then reduces these programs to a specific number of components using n_components. This step removes redundant or near-identical programs by selecting the least correlated individuals from the hall of fame.

In GP, a mathematical formula is represented as a syntax tree or a list-style expression (similar to LISP S-expressions).

Key concepts include:

• Primitive Set: The collection of all available components used to build programs. This consists of Terminals and Functions.
• Terminals: The leaves of the syntax tree, which include variables (e.g., $X_0, X_1$) and constants (e.g., $3.0, 0.5$).
• Functions: The interior nodes of the tree. Each function has an arity, which is the number of arguments it accepts (e.g., add has an arity of 2, while abs has an arity of 1).

In gplearn, the available function set is controlled via the function_set argument during estimator initialization.

To prevent programs from breaking due to mathematical errors (like division by zero), gplearn uses protected functions. These ensure that even if an invalid operation occurs, the function returns a valid numerical value so the rest of the tree can be evaluated.

Built-in protections include:

• Division: If the denominator is between -0.001 and 0.001, it returns 1.0.
• Square Root: Returns the square root of the absolute value of the argument.
• Log: Returns the logarithm of the absolute value of the argument; if the value is less than 0.001, it returns 0.0.
• Inverse: If the argument is between -0.001 and 0.001, it returns 0.0.

When defining custom functions via functions.make_function, the factory function performs basic checks to help you guard against these common invalid operations.

Because gplearn follows the scikit-learn API design, it is fully compatible with the existing scikit-learn ecosystem. You can use gplearn estimators directly within:

• scikit-learn pipelines: Use sklearn.pipeline.Pipeline to chain preprocessing steps with gplearn models.
• Grid search: Use sklearn.model_selection.GridSearchCV or RandomizedSearchCV to tune the many evolution-related parameters available in gplearn estimators.

The SymbolicClassifier evolves programs similarly to the SymbolicRegressor. The key difference is that the program's numeric output is passed through a sigmoid function to transform it into class probabilities.

• A negative output predicts one class.
• A positive output predicts the other.

Note that the sigmoid function is not included when calculating program depth or length, so bloat reduction measures behave identically to those in the regressor.

gplearn provides three primary estimators tailored for different machine learning tasks:

• SymbolicRegressor: Used for regression tasks.
• SymbolicClassifier: Used for binary classification tasks.
• SymbolicTransformer: Used for automated feature engineering. While designed primarily for regression problems, it can also be used for binary classification.

gplearn implements Genetic Programming for symbolic regression using a scikit-learn compatible API. You can use the SymbolicRegressor to find mathematical expressions that describe the relationship between independent variables (X) and dependent variable targets (y).

To get started, use the standard fit and predict pattern:

est = SymbolicRegressor()
est.fit(X_train, y_train)
y_pred = est.predict(X_test)

Ensure you have a recent version of scikit-learn (which includes numpy and scipy) installed before proceeding.

To install gplearn to your current environment, use:

pip install gplearn

To install gplearn to your user home directory, use:

pip install --user gplearn