DOCSEARCHbyAlgolia
Submit Docs

crepes Python Package

repository·main·Indexed 20 days ago

https://github.com/henrikbostrom/crepes

A Python package for conformal prediction that wraps standard classifiers and regressors to provide well-calibrated p-values, prediction sets, and intervals with coverage guarantees. It supports standard, normalized, and Mondrian conformal predictors, as well as Conformal Predictive Systems (CPS) for producing cumulative distribution functions. The library includes the crepes.martingales module for testing the exchangeability assumption and crepes.extras for difficulty estimation and categorization.

Tokens
18.6K
Snippets
58
Records
75
Agent score
67%

What's inside crepes

  1. Overview of crepes

    main

    crepes is a Python package designed for conformal prediction. It allows you to build conformal classifiers, regressors, and predictive systems on top of any standard classifier or regressor.

    Key capabilities include:

    • Converting original predictions into well-calibrated p-values and cumulative distribution functions (CDFs).
    • Generating prediction sets and intervals with coverage guarantees.
    • Implementing standard and Mondrian conformal classifiers.
    • Implementing standard, normalized, and Mondrian conformal regressors and predictive systems.

    Extensibility:

    • You can provide your own functions for computing difficulty estimates, non-conformity scores, and Mondrian categories.
    • The crepes.extras module provides standard options for these computations.
    • The crepes.martingales module provides classes for testing the exchangeability assumption.

  2. Overview of crepes functionality

    main

    crepes is a Python package used to implement conformal classifiers, regressors, and predictive systems. It works on top of any standard classifier or regressor to transform original predictions into:

    • Well-calibrated p-values and cumulative distribution functions (CDFs).
    • Prediction sets and intervals with coverage guarantees.

    Supported implementations include:

    • Standard and Mondrian conformal classifiers.
    • Standard, normalized, and Mondrian conformal regressors.
    • Predictive systems.

    Users can provide custom functions for difficulty estimates, non-conformity scores, and Mondrian categories. For standard implementations of these, use the crepes.extras module. To test the exchangeability assumption, use the crepes.martingales module.

  3. Core API components of crepes

    main

    The crepes module provides several classes for conformal prediction, categorized into classifiers, regressors, and predictive systems:

    • Classifiers: Use WrapClassifier or ConformalClassifier to perform conformal prediction for classification tasks.
    • Regressors: Use WrapRegressor or ConformalRegressor to perform conformal prediction for regression tasks.
    • Predictive Systems: Use ConformalPredictiveSystem to manage more complex predictive workflows.

  4. Use betting functions in crepes.martingales

    main

    The crepes.martingales module provides several classes for implementing different betting strategies (martingales) and functions for statistical analysis.

    Available betting strategy classes include:

    • EpsilonBettingFunction: Implements epsilon-based betting.
    • StepBettingFunction: Implements step-based betting.
    • SimpleJumper: A simple jumping strategy.
    • SleeperStayer: A strategy that stays in a 'sleeper' state.
    • SleeperDrifter: A strategy that drifts while in a 'sleeper' state.
    • CompositeMartingale: Allows for combining multiple martingale strategies.

  5. Enable semi-online calibration for conformal predictors

    main

    Standard inductive conformal predictors produce non-independent p-values. To make them independent, you can use semi-online conformal predictors. This updates the calibration set immediately after each prediction using the true label.

    To enable this, set online=True in .predict_p(), .evaluate(), or .predict_int() and provide the true labels y_test.

    # Semi-online p-values
rf_classcond.predict_p(X_test, y_test, online=True)

# Semi-online evaluation
rf_classcond.evaluate(X_test, y_test, confidence=0.99, online=True)

  6. Use Mondrian and Class-Conditional conformal classifiers

    main

    You can control error levels across different groups of objects using Mondrian conformal classifiers or class-conditional classifiers.

    Mondrian Conformal Classifiers

    Provide a categorization function or a MondrianCategorizer (from crepes.extras) to the mc argument in the .calibrate() method. This divides the object space into non-overlapping categories.

    Class-Conditional Conformal Classifiers

    A special case of Mondrian classifiers where categories are formed by the true labels. Enable this by setting class_cond=True in .calibrate().

    Semi-Online Calibration

    To make p-values independent (useful for inductive predictors), enable online calibration by setting online=True in .predict_p(), .predict_set(), or .evaluate(). This requires providing the true labels y_test.

    # Mondrian Conformal Classifier
rf_mond = WrapClassifier(rf.learner)
rf_mond.calibrate(X_cal, y_cal, mc=rf_mond.predict)

# Class-Conditional Conformal Classifier
rf_classcond = WrapClassifier(rf.learner)
rf_classcond.calibrate(X_cal, y_cal, class_cond=True)

# Semi-online prediction (requires y_test)
rf_classcond.predict_p(X_test, y_test, online=True)

  7. Implement Mondrian and Class-Conditional conformal classifiers

    main

    You can control error levels across different groups using Mondrian conformal classifiers or class-conditional classifiers:

    • Mondrian Conformal Classifier: Provide a categorization function or a MondrianCategorizer (from crepes.extras) to the mc argument in .calibrate().
    • Class-Conditional Conformal Classifier: A special case of Mondrian where categories are formed by true labels. Enable this by setting class_cond=True in .calibrate().

    Note: When using a categorization function, the same logic is applied to test objects during prediction.

    # Mondrian using a prediction function
rf_mond = WrapClassifier(rf.learner)
rf_mond.calibrate(X_cal, y_cal, mc=rf_mond.predict)

# Class-conditional conformal classifier
rf_classcond = WrapClassifier(rf.learner)
rf_classcond.calibrate(X_cal, y_cal, class_cond=True)

  8. Use Conformal Predictive Systems (CPS)

    main

    Conformal Predictive Systems (CPS) produce cumulative distribution functions (conformal predictive distributions) instead of just intervals. To enable this, pass cps=True to the .calibrate() method of a WrapRegressor or WrapClassifier.

    With a CPS, you can:

    • Get percentiles: rf.predict_percentiles(X_test, higher_percentiles=[90, 95, 99])
    • Get the full distribution: rf.predict_cpds(X_test)
    • Get prediction intervals (standard behavior for CPS):
      • For standard/normalized CPS: returns a 2D array.
      • For Mondrian/semi-online CPS: returns a vector containing one CPD per instance.
    # Enable CPS during calibration
rf.calibrate(X_cal, y_cal, de=de, mc=mc_pred, cps=True)

# Get specific percentiles
percentiles = rf.predict_percentiles(X_test, higher_percentiles=[90, 95, 99])

# Get full CPDs
cpds = rf.predict_cpds(X_test)

  11. Use WrapRegressor for conformal regression

    main

    For regression tasks, wrap a scikit-learn regressor using WrapRegressor.

    1. Fit: .fit(X_prop_train, y_prop_train).
    2. Calibrate: .calibrate(X_cal, y_cal).
    3. Predict Intervals: .predict_int(X_test, confidence=0.99) returns a NumPy array where each row contains [lower_bound, upper_bound].
      • Use y_min (e.g., y_min=0) to cut intervals at a specific value to exclude impossible results.
    4. Normalized Intervals: Use a DifficultyEstimator (from crepes.extras) to assign wider intervals to harder objects. Pass the fitted estimator to .calibrate(..., de=de).
    from sklearn.ensemble import RandomForestRegressor
from crepes import WrapRegressor
from crepes.extras import DifficultyEstimator

# Setup regressor
rf = WrapRegressor(RandomForestRegressor())
rf.fit(X_prop_train, y_prop_train)

# Optional: Difficulty estimation for normalized intervals
de = DifficultyEstimator()
de.fit(X_prop_train, y=y_prop_train)

# Calibrate with difficulty estimator
rf.calibrate(X_cal, y_cal, de=de)

# Predict intervals with a lower bound constraint
intervals = rf.predict_int(X_test, y_min=0)

  12. Generate conformal regressors with WrapRegressor

    main

    For regression tasks, wrap a scikit-learn regressor using WrapRegressor.

    1. Fit the wrapped learner: rf.fit(X_prop_train, y_prop_train).
    2. Calibrate: rf.calibrate(X_cal, y_cal).
    3. Predict intervals: rf.predict_int(X_test, confidence=0.99).

    Advanced Regression Features

    • Constraint Clipping: Use y_min to ensure intervals do not include impossible values (e.g., rf.predict_int(X_test, y_min=0)).
    • Normalized Intervals: Use a DifficultyEstimator (from crepes.extras) to assign wider intervals to more difficult objects. Pass the fitted estimator to .calibrate(..., de=de).
    • Mondrian Regressors: Use a MondrianCategorizer via the mc argument in .calibrate() to divide the space into categories for more robust regression.
    • Semi-Online: Use online=True in prediction methods while providing y_test to ensure independent errors.
    from sklearn.ensemble import RandomForestRegressor
from crepes import WrapRegressor
from crepes.extras import DifficultyEstimator

# Setup
rf = WrapRegressor(RandomForestRegressor())
rf.fit(X_prop_train, y_prop_train)
rf.calibrate(X_cal, y_cal)

# Basic prediction intervals
intervals = rf.predict_int(X_test, confidence=0.99)

# Normalized intervals using DifficultyEstimator
de = DifficultyEstimator()
de.fit(X_prop_train, y=y_prop_train)
rf.calibrate(X_cal, y_cal, de=de)
intervals_norm = rf.predict_int(X_test, y_min=0)