rfpimp

Features in machine learning are rarely independent. To identify if a feature $x$ is dependent on other features, you can train a model using $x$ as the target and all other features as predictors.

Because random forests provide an out-of-bag (OOB) error estimate, the feature dependence functions in this project use random forest models to calculate this. A high $R^2$ prediction error indicates that feature $x$ is highly dependent on the other features.

Mean Decrease in Impurity (Default scikit-learn)

Scikit-learn's default strategy measures how much a feature reduces uncertainty (classifiers) or variance (regressors) within decision trees. While fast, it is unreliable when predictor variables vary in scale or number of categories.

Permutation Importance (rfpimp)

This method records a baseline performance score (accuracy or $R^2$) on a validation set. It then permutes the values of a single feature and recomputes the score. The importance is the difference between the baseline and the drop in performance. It is more computationally expensive but provides more reliable results.

To use reliable permutation and drop-column importance measures with scikit-learn models, install the rfpimp package using pip.

Note: While the name suggests it is only for Random Forests, it works with any scikit-learn model.

pip install rfpimp

A method to estimate feature importance is to measure the drop in the model's Out-of-Bag (OOB) score when a specific column is removed from the training set.

This approach involves cloning the base model, fitting it on the full dataset to establish a baseline OOB score, and then iteratively fitting clones of the model on datasets where one feature is dropped at a time. The importance is defined as baseline_oob_score - dropped_column_oob_score.

def dropcol_importances(rf, X_train, y_train):
    rf_ = clone(rf)
    rf_.random_state = 999
    rf_.fit(X_train, y_train)
    baseline = rf_.oob_score_
    imp = []
    for col in X_train.columns:
        X = X_train.drop(col, axis=1)
        rf_ = clone(rf)
        rf_.random_state = 999
        rf_.fit(X, y_train)
        o = rf_.oob_score_
        imp.append(baseline - o)
    return np.array(imp)

To use plot_importances(), data must be formatted into a pandas DataFrame where the index represents the feature names and the values represent the importance scores, sorted in descending order.

While rfpimp provides the plotting logic, you often need to transform raw CSV data or model outputs into this specific format. A common pattern is to create a helper function (like mkdf in the examples) that takes column names and importance values, constructs a DataFrame, sets the index, and sorts the values.

import pandas as pd

def mkdf(columns, importances):
    I = pd.DataFrame(data={'Feature':columns, 'Importance':importances})
    I = I.set_index('Feature')
    I = I.sort_values('Importance', ascending=False)
    return I

The importances() function calculates permutation importance by measuring the drop in a model's baseline performance (accuracy for classifiers or $R^2$ for regressors) when a feature's column values are permuted. This is more reliable than the default scikit-learn 'mean decrease in impurity' method.

Parameters:

• model: A trained scikit-learn model.
• X: The feature set (test set or out-of-bag samples).
• y: The target values.
• n_samples: (Optional) Number of samples to use for permutation.

from rfpimp import *
# ... setup model, X_test, y_test ...
imp = importances(rf, X_test, y_test)

After calculating importances using importances(), you can visualize the results using plot_importances(). The returned object provides a .view() method to display the plot.

imp = importances(rf, X_test, y_test)
viz = plot_importances(imp)
viz.view()

from rfpimp import *
# ... setup model, X_test, y_test ...
imp = importances(rf, X_test, y_test)
viz = plot_importances(imp)
viz.view()

When using the features argument in importances(), you can define complex groupings. A feature can be included in multiple groups, which allows you to see how a specific variable contributes to different conceptual clusters of features.

# Features can be duplicated in multiple groups
features = [['latitude', 'longitude'],
            ['price_to_median_beds', 'beds_baths', 'beds_per_price', 'bedrooms'],
            ['price','beds_per_price','bedrooms']]

I = importances(rf, X_test, y_test, features=features)
viz = plot_importances(I, vscale=1.2)

You can extract feature importances directly from a fitted scikit-learn RandomForestClassifier using the .feature_importances_ attribute. This attribute typically represents the average reduction in impurity (e.g., Gini importance) brought by a feature.

Note that built-in importances can sometimes be biased towards high-cardinality features. To visualize these, you can use the plot_importances function from rfpimp after converting the results into a pandas DataFrame.

# Assuming rf is a fitted RandomForestClassifier
# and mkdf is a helper to create a sorted DataFrame
I = mkdf(X_train.columns, rf.feature_importances_)
viz = plot_importances(I, imp_range=(0, .4), title="Feature importance via avg drop in variance (sklearn)")
viz.show()

Calculate the correlation matrix between all pairs of features and visualize it as a heatmap.

Parameters for feature_corr_matrix:

• df: Dataframe containing features.
• method: Correlation method, either 'spearman', 'pearson', a callable, or a numpy array.

Parameters for plot_corr_heatmap:

• df: Dataframe to correlate.
• color_threshold: Absolute value threshold for background coloring.
• method: Correlation method (default 'spearman').

from rfpimp import plot_corr_heatmap
# ... df_train defined ...
viz = plot_corr_heatmap(df_train, save='/tmp/corrheatmap.svg', 
                      figsize=(7,5), label_fontsize=13, value_fontsize=11)
viz.view()