MAPIE Documentation

MAPIE implements three primary methods for risk control, which allow you to adjust model parameters to ensure a specific risk level (like precision or recall) is maintained on unseen data.

Terminology Note

• alpha ($\alpha$) is the target risk level (equivalent to 1 - confidence_level).
• calibrate and calibration are used interchangeably with conformalize and conformalization.

Comparison of Methods

Application in Multi-label Classification

• For recall control: Use CRC or RCPS.
• For precision control: Use LTT.

RCPS is a method for computing prediction sets that control risk with high probability. It requires a set-valued function $\mathcal{T}_{\hat{\lambda}}$ that is nested (larger $\lambda$ results in larger sets) and a loss function $L$ that is monotonic (larger sets result in lower loss).

Guarantee

$\mathbb{P}(R(\mathcal{T}_{\hat{\lambda}}) \leq \alpha) \geq 1 - \delta$

Available Bounds for Upper Confidence Bound $\hat{R}^+(\lambda)$

• Hoeffding Bound: $\hat{R}_{\text{Hoeffding}}^+(\lambda) = \hat{R}(\lambda) + \sqrt{\frac{1}{2n}\log\frac{1}{\delta}}$
• Bernstein Bound: $\hat{R}_{\text{Bernstein}}^+(\lambda) = \hat{R}(\lambda) + \hat{\sigma}(\lambda)\sqrt{\frac{2\log(2/\delta)}{n}} + \frac{7\log(2/\delta)}{3(n-1)}$
• Waudby-Smith–Ramdas: Recommended for bounded losses.

CRC controls any monotone and bounded loss function. It provides a guarantee on the expected risk:

$$\mathbb{E}\left[L_{n+1}(\hat{\lambda})\right] \leq \alpha$$

It finds the optimal parameter $\hat{\lambda}$ by solving for the smallest $\lambda$ such that the empirical risk (adjusted for the sample size) stays below the target $\alpha$.

Several statistical metrics can be used to evaluate how well a calibration curve follows the ideal diagonal line (perfect calibration):

• Kolmogorov-Smirnov (KS) Statistics: Tests if the calibration curve deviates significantly from the ideal diagonal.
• Kuiper Statistics: Similar to KS, but specifically designed to capture both positive and negative deviations from perfect calibration.
• Spiegelhalter Statistics: Used to test the overall calibration of predicted probabilities.
• Cumulative Differences: Calculates the cumulative differences between sorted true values and prediction scores.

LTT is a versatile method that can control any loss, including non-monotonic ones, by using multiple hypothesis testing.

Process

1. For each $\lambda_j$ in a discrete set $\Lambda$, estimate the risk on calibration data.
2. Associate a hypothesis $\mathcal{H}_j: R(\lambda_j) > \alpha$.
3. Compute p-values (e.g., using Hoeffding-Bentkus).
4. Apply Family-Wise Error Rate (FWER) control, such as the Bonferroni correction.

Guarantee

$\mathbb{P}(R(\mathcal{T}_{\lambda}) \leq \alpha) \geq 1 - \delta$ for all $\lambda \in \hat{\Lambda}$.

The workflow has been restructured to separate model training from the conformalization (calibration) step.

Split Conformal Workflow

1. Data splitting: Use train_conformalize_test_split() to prepare your data.
2. Model training: Call .fit(X_train, y_train) on your chosen regressor or classifier.
3. Conformalization: Call .conformalize(X_conf, y_conf) to calibrate the model.
4. Prediction: Use .predict_interval() (for regression) or .predict_set() (for classification).

Cross Conformal Workflow

For cross-conformal techniques, steps 2 and 3 are combined into a single call:

1. Data splitting: Use train_conformalize_test_split().
2. Training & Conformalization: Call .fit_conformalize(X_train, y_train, X_conf, y_conf).
3. Prediction: Use .predict_interval() or .predict_set().

The Jackknife+ method is a robust way to estimate prediction intervals that accounts for model variability by using leave-one-out predictions on the new test point.

Key Properties:

• Guarantee: It guarantees a coverage level of $1-2\alpha$ for a target of $1-\alpha$ without a priori assumptions on the data distribution or the predictive model.
• Mechanism: It uses each leave-one-out prediction on the new test point to account for variability.

Warning: Jackknife methods (including Jackknife+) require running as many simulations as training points, which can be computationally prohibitive for large datasets.

The method replaces the fixed quantile $S^*$ used in standard split conformal prediction with a function $\hat{g}{S(X{n+1}, y)}(X_{n+1})$ that estimates conditional quantiles of the conformity scores.

In the finite-dimensional setting, the process follows three steps:

1. Choose a class of functions $\mathcal{F}$: A common approach is to use linear functions defined by a user-provided feature map $\Phi: \mathcal{X} \to \mathbb{R}^d$.
2. Optimize the function: Find the best function in the class by solving an optimization problem (in practice, a dual formulation is used) that minimizes the pinball loss over the calibration set.
3. Compute intervals: Use the optimized function $\hat{g}S$ to define the prediction set: $$\hat{C}(X{n+1}) = { y : S(X_{n+1}, : y) \leq \hat{g}{S(X{n+1}, y)}(X_{n+1}) }$$

Standard thresholding (e.g., choosing a probability threshold for binary classification) based on validation data offers no statistical guarantees on new, unseen data.

Risk control solves this by adjusting a model parameter $\lambda$ so that a given risk stays below a desired level $\alpha$ with high probability (or in expectation) on unseen data.

Mathematical Guarantees

• CRC (Conformal Risk Control): Requires exchangeable data and provides an expectation guarantee: $\mathbb{E}(R) \leq \alpha$.
• RCPS (Risk-Controlling Prediction Sets) and LTT (Learn Then Test): Require i.i.d. data and provide a probability guarantee: $\mathbb{P}(R \leq \alpha) \geq 1 - \delta$, where $\delta$ is the confidence level.