DOCSEARCHbyAlgolia
Submit Docs

Probatus

repository·main·Indexed 41 days ago

https://github.com/ing-bank/probatus

A Python package for validating regression and multiclass classification models and their underlying data. It provides tools for SHAP-based model interpretation, dataset similarity analysis via resemblance modelling, and cross-validated Recursive Feature Elimination using the ShapRFECV class.

Tokens
10.5K
Snippets
35
Records
45
Agent score
100%

What's inside Probatus

  1. Overview of Probatus

    main

    Probatus is a Python library designed to analyze binary classification models and the data used to develop them. Its primary capabilities include:

    • Assessing metric stability.
    • Analyzing differences between two data samples (e.g., detecting shifts between training and testing splits).

  2. What is Probatus used for?

    main

    Probatus is a toolkit designed for Data Scientists to analyze key aspects of building classification models. It focuses on three main pillars:

    1. Model input: Evaluating dataset quality and preparation for modeling.
    2. Model performance: Assessing model quality and the stability of results.
    3. Model interpretation: Understanding the decision-making processes of models.

    The library is designed to be highly usable and works seamlessly with scikit-learn compatible models.

  3. Overview of Probatus features

    main

    Probatus is a Python package designed to validate regression and multiclass classification models, as well as the data used to develop them.

    Key functional areas include:

    • Model Interpretation: Use probatus.interpret for SHAP-based model interpretation tools.
    • Dataset Comparison: Use probatus.sample_similarity to compare two datasets (e.g., comparing a train set against an out-of-time test set) using resemblance modelling.
    • Feature Selection: Use probatus.feature_elimination.ShapRFECV for cross-validated Recursive Feature Elimination powered by SHAP feature importance.

  4. Understand Sample Similarity using the Resemblance Model

    main

    The Resemblance Model is used to understand how different two multivariate datasets ($X_1$ and $X_2$) are from a distributional perspective.

    How it works:

    1. Assign label 0 to rows from $X_1$ and label 1 to rows from $X_2$.
    2. Train a binary classification model to predict the dataset label for a given row.
    3. Evaluate the model using AUC (Area Under the Curve):
      • An AUC significantly higher than 0.5 indicates that the two datasets have different distributions (the model can easily distinguish between them).
      • An AUC near 0.5 suggests the datasets are similar and the model cannot distinguish them better than random chance.
    4. Analyze feature importance to identify which specific features contribute most to the difference between the datasets.

  5. Interpret models using SHAP with Probatus

    main
    Probatus provides a convenience wrapper for the SHAP library to facilitate model interpretation. This module is designed to implement multiple visualization plots specifically for tree-based and linear models, making it easier to understand feature importance and model behavior.

  8. How Resemblance Models work for Sample Similarity

    main

    A Resemblance Model is used to understand how two multivariate samples (X1 and X2) differ from each other. This is useful for detecting distribution shifts, such as comparing an In-Time Train set to an Out-Of-Time Test set.

    The Process

    1. Labeling: The model assigns label 0 to rows from X1 and label 1 to rows from X2.
    2. Training: The combined data is shuffled and split into a Train split and a Test split. A binary classifier is then fitted on the Train split.
    3. Evaluation: The model is evaluated on both Train and Test sets.

    Interpreting Results

    • AUC Scores: A Test AUC significantly above 0.5 indicates that the model can successfully distinguish between the two datasets, implying a change in distribution.
    • Model Fit: It is critical that the model does not overfit or underfit. Ensure that the Train AUC is not significantly higher than the Test AUC to avoid drawing wrong conclusions from the feature importance.
    • Feature Importance: Once trained, the model can be inspected to identify which features or interactions differ between the two samples.

  9. Detecting data or target distribution shift with SHAP

    main

    You can detect shifts in data or target distributions by comparing the SHAP metrics between your training and testing sets.

    If there is a significant difference between mean_abs_shap_value_train and mean_abs_shap_value_test, or between mean_shap_value_train and mean_shap_value_test for specific features, it indicates a potential shift in the data or target distribution. You can visualize these differences by generating separate summary plots for the "train" and "test" sets.

  10. Configure Cross-Validation splitters

    main

    When performing feature elimination or other CV-based tasks, you can provide a CV object. A CV can be:

    • An integer (number of folds).
    • A CVSplitter protocol object implementing a .split() method.
    • An Iterable of tuples containing (train_indices, test_indices) as IndexArray (numpy integer arrays).
    import numpy as np

# Example of providing manual CV indices
custom_cv = [
    (np.array([0, 1, 2]), np.array([3])),
    (np.array([0, 1, 3]), np.array([2])),
]

  11. Ensure static inputs for reproducibility

    main

    Reproducibility depends on the stability of the inputs passed to probatus modules. Follow these guidelines for common inputs:

    Static Dataset

    Ensure the input dataset X and y do not change. If generating synthetic data (e.g., using sklearn.datasets.make_classification), always set the random_state.

    Static Data Splits

    • Train/Test Splits: When using sklearn.model_selection.train_test_split, set the random_state.
    • Cross-Validation (CV): If you pass an integer to the cv parameter in probatus, the module's internal random_state will handle the splitting. However, if you pass a custom CV generator object (like StratifiedKFold), you must set the random_state within that object itself.

    Static Classifier

    • If the module requires a not-fitted classifier, set the random_state on the classifier instance.
    • If the classifier must be pre-fitted, ensure the training process itself is reproducible by setting the model's random_state.

    Static Hyperparameter Tuning

    When using modules that perform optimization (e.g., ShapRFECV), ensure the search object (like RandomizedSearchCV) has a fixed random_state. This ensures the same parameter permutations are explored in every round. If the search space generation is stochastic, set its random_state as well.

    from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

# 1. Static train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# 2. Static CV settings
cv2 = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# 3. Static classifier
model = RandomForestClassifier(random_state=42)

# 4. Static search CV for hyperparameter tuning
param_grid = {"n_estimators": [5, 7, 10], "max_leaf_nodes": [3, 5, 7, 10]}
search = RandomizedSearchCV(model, param_grid, n_iter=1, random_state=42)

  12. Handle outliers in DependencePlotter plots

    main

    When your dataset contains outliers, the SHAP dependence plots and histograms may become unreadable. You can mitigate this by using the min_q and max_q parameters in the .plot() method to filter data points based on quantiles.

    • min_q: The minimum quantile threshold. Data points below this quantile are excluded.
    • max_q: The maximum quantile threshold. Data points above this quantile are excluded.

    For example, to remove the top 1% of extreme values (outliers), set max_q=0.99.

    # Remove outliers by setting the maximum quantile to 0.99
fig = tdp.plot(feature="sepal_length", figsize=(7, 4), max_q=0.99)