PowerSHAP: Feature Selection using Statistical Significance of SHAP Values

PowerShap selects features by comparing their impact on model predictions against the impact of a known random feature.

1. Iteration: The algorithm trains multiple models on different data subsets. In each iteration, a random uniform feature is added to the dataset.
2. Shapley Calculation: For each model, it calculates the absolute Shapley values. For multi-output or multi-class tasks, it uses the maximum across outputs. These values are averaged per feature to represent impact.
3. Statistical Testing: After all iterations, each feature's impact array is compared to the random feature's impact array using a percentile formula to generate a p-value.
4. Selection: Features with a p-value below a specific threshold (default is 0.01) are retained.

Automatic Mode

To avoid manual hyperparameter tuning for the number of iterations and the threshold, PowerShap uses an automatic mode by default:

• It starts with 10 iterations.
• It calculates the effect size and statistical power (using a student-t power test) for each feature.
• It determines the required iterations to meet a predefined power requirement (default is 0.99, representing a false positive probability of 0.01).
• It continues executing additional iterations until the required power is achieved.

PowerShap provides two advanced modes for optimizing the feature selection process:

Automatic Mode

When automatic=True, the selector doesn't use a fixed number of power_iterations. Instead:

1. It performs an initial run with 10 iterations.
2. It calculates the required number of iterations needed to reach the power_req_iterations threshold for the given power_alpha.
3. It restarts the process using this calculated number of iterations.
4. If the required iterations exceed limit_automatic, it incrementally adds limit_incremental_iterations and re-evaluates (up to limit_recursive_automatic times).

Force Convergence

When force_convergence=True (typically used in high-dimensional datasets), PowerShap performs a recursive selection:

1. It identifies significant features.
2. It removes those features from the dataset.
3. It reruns the selection process on the remaining features.
4. This continues until no more significant features are found or limit_convergence_its is reached.

This ensures that the selection is robust and that features are not being selected simply because they are masking other important variables.

Benchmark results are often stored in Markdown tables. To process them for visualization, you can use pandas.read_table with a pipe | separator.

Parsing Logic:

1. Load: Use pd.read_table(path, sep="|", ...).
2. Clean: Strip whitespace from column names and handle empty columns.
3. Extract Statistics: If metrics are stored as value (std_dev) (e.g., 0.85 (0.02)), use string manipulation to separate the mean and the standard deviation into two distinct columns (metric and metric_std).
4. Normalize Method Names: Map various string patterns (e.g., Automatic default powershap) to canonical names (e.g., powershap) for consistent plotting.

import pandas as pd
import re

# Load markdown table
df_t = pd.read_table("../../data/dataset.md", sep="|", header=0, index_col=1, skipinitialspace=True).dropna(axis=1, how='all').iloc[1:]

# Extract mean and standard deviation from 'Value (Std)' format
for col in ['AUC', 'MCC', 'ACC']:
    df_t[col+"_std"] = 0
    mask = df_t[col].str.contains("\(\\)")
    # Extract value inside parentheses
    df_t.loc[mask, col+"_std"] = [pd.to_numeric(s[s.find("(")+1:s.find(")")]) for s in df_t.loc[mask, col].values]
    # Extract mean value before the parenthesis
    df_t[col] = [pd.to_numeric(s[:s.find(" ")]) for s in df_t[col].values]

When initializing PowerShap, you can tune several parameters to control the selection process:

Once benchmark data is parsed, use Matplotlib bar plots to compare methods across different datasets and evaluation sets.

For Classification:

• Use metrics like AUC, MCC, ACC, Recall, F1, or Precision.
• Use yerr in plt.bar() to represent the standard deviation extracted during parsing.
• Create a grid of subplots where each subplot represents a dataset and an Eval set.

For Regression:

• Use metrics like R², RMSE, MAE, or ME.
• Follow a similar subplot grid pattern to classification benchmarks.

import matplotlib.pyplot as plt

metric = "AUC"
f, axs = plt.subplots(2, 3, figsize=(15, 6))

# Iterate through unique evaluation sets and datasets to fill the grid
for i, evalset in enumerate(end_class_df["Eval set"].unique()):
    for j, dataset in enumerate(end_class_df.dataset.unique()):
        current_df = end_class_df[(end_class_df.dataset == dataset) & (end_class_df["Eval set"] == evalset)]
        axs[i][j].bar(
            current_df.Method.values, 
            current_df[metric].values, 
            yerr=current_df[metric+"_std"].values, 
            capsize=10
        )
        axs[i][j].set_title(f"{dataset} : {evalset}")