autofeat

Autofeat is a Python library designed to improve linear model performance by automating the generation of non-linear features and selecting a robust subset of them.

Key Features

• Automated Feature Generation and Selection: Automatically creates and selects features to enhance linear models.
• Improved Performance and Interpretability: Increases prediction accuracy while maintaining the transparency of linear models.
• Seamless Integration: Fully compatible with scikit-learn pipelines.

Use Cases

• Supervised learning where model transparency is required.
• Feature selection in large datasets to automate variable discovery.
• Non-linear feature discovery to leverage complex relationships without increasing model complexity.

Automated feature engineering can generate a very large feature matrix. To manage memory and computational load, use the following strategies:

• Limit Engineering Scope: Use the feateng_cols parameter to specify only the columns you expect to be most valuable for engineering. This significantly reduces the number of generated features.
• Restrict Transformations: Limit the transformations parameter to only those specific transformations that are relevant to your data type.
• Subsample Training Data: Subsample the dataset used during the fit() process to limit memory requirements. Once the model is fit, you can still call transform() on your full dataset to generate only the selected features.

You can install the autofeat library directly from PyPI using pip. This library provides sklearn-compatible linear prediction models with automated feature engineering and selection.

pip install autofeat

For more information, you can access the following resources:

• Example Notebooks: Located in the /notebooks/ directory of the repository.
• Official Documentation: https://cod3licious.github.io/autofeat
• Research Paper: https://arxiv.org/abs/1901.07329
• PyData Talk: Watch on YouTube

If you provide a units dictionary to AutoFeatModel, the model can use the Pi Theorem to generate physically meaningful dimensionless features.

Requirements:

• apply_pi_theorem must be True (default).
• units must be a dictionary mapping column names to strings compatible with pint (e.g., {'velocity': 'm/s', 'time': 's'}).
• It is assumed that all features are of comparable magnitude. Scale your variables before passing them to AutoFeat if they differ significantly in scale (e.g., meters vs millimeters).

Generated features from the Pi Theorem will be prefixed with PT.

# Example of providing units
units_config = {
    'speed': 'm/s',
    'time': 's',
    'distance': 'm'
}

model = AutoFeatModel(units=units_config, apply_pi_theorem=True)
# The model will now attempt to generate features like distance/time (speed)

AutoFeat's multi-step feature selection process relies on multiple rounds of L1-regularized linear models. To be effective, these models must balance speed with the ability to select reliable features (high precision and high recall).

For Regression tasks, AutoFeat uses LassoLarsCV because it provides a good trade-off between execution speed and the quality of selected features compared to other models like ElasticNet, Lasso, or OrthogonalMatchingPursuit.

For Classification tasks, AutoFeat uses sklearn.linear_model.LogisticRegressionCV because it is significantly faster than alternatives like svm.LinearSVC (even when using grid search).

You can use the following patterns to generate synthetic data for testing feature selection performance:

Regression Problem

Create a target variable as a linear combination of random features and add normally distributed noise:

X = np.random.randn(n_train, n_feat_noise + n_feat_true)
true_features = np.random.permutation(n_feat_noise + n_feat_true)[:n_feat_true]
y = X[:, true_features].sum(axis=1)

# Add noise
y -= y.mean()
y /= y.std()
y = (1 - noise) * y + noise * np.random.randn(len(y))

Classification Problem

Transform a regression problem into a classification problem by thresholding the target and randomly flipping labels to introduce noise:

y = np.array(y > y.mean(), dtype=int)
flip_idx = np.random.permutation(len(y))[: int(np.ceil(len(y) * noise))]
y[flip_idx] -= 1
y = np.abs(y)

The AutoFeatLight model provides a streamlined version of the feature engineering process. It is designed for faster execution by performing only minimal operations:

• Feature Selection: Removes zero-variance and redundant features.
• Feature Engineering: Performs simple product and ratio operations on features.
• Scaling: Applies a power transform to make features more normally distributed.