ppscore

The PPS implementation identifies specific data scenarios to either optimize computation or signal unsupported inputs.

Valid scores (Optimized to 0 or 1)

These cases return a score without fitting a model:

• feature_is_id: Categoric feature where all categories appear only once. Score: 0.
• target_is_id: Categoric target where all categories appear only once. Score: 0.
• target_is_constant: Target column has only one unique value. Score: 0.
• predict_itself: Feature and target columns are identical. Score: 1.

Invalid scores (Returns invalid_score)

These cases indicate the PPS cannot be calculated:

• target_is_datetime: Target is a datetime type (Solution: convert to string).
• target_data_type_not_supported: Target has an unsupported data type.
• empty_dataframe_after_dropping_na: No rows remain after dropping NaNs (Solution: handle missing values first).
• unknown_error: An unexpected error occurred. This is only reported if catch_errors=True. To debug, set catch_errors=False to see the actual exception.

The PPS is a bivariate metric that measures how well a single feature can predict a target column. Key aspects of the calculation include:

• No Interaction Effects: Unlike feature importance, PPS calculates scores for each feature independently. It does not account for interactions between multiple features.
• Cross-Validation: Scores are calculated using the test sets of a $K$-fold cross-validation (default is 4). For classification, stratifiedKFold is used; for regression, KFold is used.
• Sampling: To optimize performance, if the dataset exceeds 5,000 rows, a random subset of 5,000 rows is used by default. This can be adjusted via the sample parameter.
• Missing Values: Rows containing missing values in either the feature or the target column are dropped before calculation.
• Reproducibility: Because the process involves random sampling and shuffling, use the random_seed parameter to ensure reproducible results.
• Error Handling: If a score cannot be calculated, the package returns an object where is_valid_score is False and the score is set to invalid_score. This prevents the entire process from crashing when scanning large datasets.

When evaluating performance metrics like the F1 score, you can establish a baseline by comparing two naive predictors: the 'most common value' and 'random guessing'.

Key observations for choosing a baseline:

• Skewed classes (2 classes): The most common value is often slightly better than a random guess.
• Skewed classes (4 classes): A random guess is often slightly better than the most common value.
• Balanced classes (2 or 4 classes): Random guessing is usually significantly better than the most common value.

Conclusion: Random values are generally preferred over the most common value as a baseline. However, the most robust baseline is the maximum of the F1 score achieved by the most common value and the F1 score achieved by random values.

import numpy as np
import pandas as pd
from sklearn.metrics import f1_score

# Example helper for most common value baseline
def f1_score_most_common(series, value):
    # Note: In practice, use the actual length of the series instead of hardcoded 1000
    return f1_score(series, np.random.choice([value], len(series)), average="weighted")

# Example helper for random baseline
def f1_score_random(series):
    return f1_score(series, series.sample(frac=1), average="weighted")

You can visualize PPS results using seaborn or other visualization libraries.

Plotting Predictors: To plot the predictive power of all features against a target, use pps.predictors and pass the resulting DataFrame to sns.barplot.

Plotting a PPS Matrix: To create a heatmap of the PPS matrix, you must first pivot the tidy DataFrame returned by pps.matrix so that columns and indices match the expected heatmap format.

Example:

import seaborn as sns
import ppscore as pps

# Plotting predictors
predictors_df = pps.predictors(df, y="y")
sns.barplot(data=predictors_df, x="x", y="ppscore")

# Plotting matrix heatmap
matrix_df = pps.matrix(df)[['x', 'y', 'ppscore']].pivot(columns='x', index='y', values='ppscore')
sns.heatmap(matrix_df, vmin=0, vmax=1, cmap="Blues", linewidths=0.5, annot=True)

import seaborn as sns
import ppscore as pps

# Plotting predictors
predictors_df = ppscore.predictors(df, y="y")
sns.barplot(data=predictors_df, x="x", y="ppscore")

# Plotting matrix heatmap
matrix_df = ppscore.matrix(df)[['x', 'y', 'ppscore']].pivot(columns='x', index='y', values='ppscore')
sns.heatmap(matrix_df, vmin=0, vmax=1, cmap="Blues", linewidths=0.5, annot=True)

Use ppscore.score(df, x, y) to calculate the Predictive Power Score for a specific feature x predicting a target y.

Key Concepts:

• The score ranges from 0 (no predictive power) to 1 (perfect predictive power).
• It is asymmetric and data-type agnostic, meaning it can detect both linear and non-linear relationships.
• A score of 0 means x cannot predict y better than a naive baseline.

Parameters:

• df: pandas.DataFrame containing the columns.
• x: str, the name of the feature column.
• y: str, the name of the target column.
• sample: int or None. Number of rows to sample to decrease calculation time. If None, no sampling occurs.
• cross_validation: int. Number of iterations for cross-validation. Higher values require more observations to avoid errors.
• random_seed: int or None. Set for reproducible results.
• invalid_score: any. Value returned if calculation is invalid (e.g., unsupported data type).
• catch_errors: bool. If True, errors are caught and reported as unknown_error. If False, exceptions are raised.

import ppscore as pps
# Calculate PPS for x predicting y
result = pps.score(df, "x", "y")