Frouros: Drift Detection for Machine Learning

The frouros.datasets module provides access to both real-world and synthetic datasets for testing and evaluating drift detection algorithms.

• Real datasets: Located in frouros.datasets.real.
• Synthetic datasets: Located in frouros.datasets.synthetic.

The frouros.detectors.concept_drift module provides algorithms designed to detect changes in the underlying distribution of data streams (concept drift). These detectors are typically used in streaming contexts to monitor model performance or data statistics over time.

Detailed implementations for streaming-specific detection can be found in the frouros.detectors.concept_drift.streaming submodule.

The frouros.detectors module is the primary entry point for accessing various detection algorithms in the library. It is organized into two main sub-modules based on the type of drift being monitored:

1. Concept Drift: Algorithms designed to detect changes in the relationship between input data and the target variable (e.g., changes in model performance).
2. Data Drift: Algorithms designed to detect changes in the underlying distribution of the input data (e.g., covariate shift).

You can explore specific implementations by navigating to the detectors/concept_drift or detectors/data_drift sub-modules.

The frouros.callbacks.batch module provides callback mechanisms designed to operate on batches of data rather than individual samples. This is useful for statistical tests or metrics that require a collection of recent observations to compute a value (e.g., detecting concept drift via permutation tests or resetting statistical tests after a batch is processed).

Available batch-oriented components include:

• PermutationTestDistanceBased: A callback for performing permutation tests based on distance metrics.
• ResetStatisticalTest: A callback used to reset the state of a statistical test.

The frouros.detectors.batch module provides algorithms for detecting data drift. Data drift detection is used to identify changes in the distribution of input data over time. The module is organized into two primary modes of operation:

• Batch detection: Algorithms designed to work on fixed datasets or chunks of data.
• Streaming detection: Algorithms designed to work on continuous data streams.

You can find specific implementations under frouros.detectors.batch.data_drift.batch and frouros.detectors.batch.data_drift.streaming.

The frouros.detectors.data_drift.streaming module provides algorithms designed to detect changes in data distributions within a streaming context. These detectors are categorized into two main approaches:

1. Distance Based: Algorithms that measure the distance between distributions (e.g., MMD).
2. Statistical Test: Algorithms that use statistical hypothesis testing to identify drift (e.g., IncrementalKSTest).

While related, these two monitoring tasks have different objectives:

• Out-of-Distribution (OOD) Detection: Focuses on identifying individual samples that fall outside the training distribution. It is typically used to detect anomalies or novel data points.
• Data Drift Detection: Concerned with identifying shifts or changes in the statistical distribution of the data over time.

Verification latency (or delay) is the time period between a model making a prediction and the availability of the ground-truth label (or target value).

• High Latency/No Labels: If labels are delayed or unavailable, you cannot use concept drift detectors because they require $y$ to function. In these scenarios, you must use data drift methods, which only require the input features $X$.
• Low Latency/Labels Available: If labels are available immediately, you can use concept drift detectors to monitor the actual relationship between features and targets.

Frouros is a specialized Python library dedicated exclusively to drift detection in machine learning systems.

Unlike multi-purpose machine learning libraries, Frouros does not provide functionalities for:

• Anomaly/outlier detection
• Adversarial detection
• Imbalance learning

It focuses solely on providing both classical and modern algorithms for detecting concept drift and data drift, designed to be compatible with any machine learning framework.

In frouros, drift is categorized into two main types based on which part of the joint distribution $P(X, y)$ changes:

• Concept Drift: A change in the conditional probability $P(y|X)$. This means the relationship between the features and the target label has changed. Concept drift methods in frouros are used to detect when the model's fundamental understanding of the problem is no longer valid.
• Data Drift: A change in the covariate distribution $P(X)$. This focuses solely on the distribution of the input features. While data drift does not guarantee model performance decay, it is a strong indicator. frouros also uses data drift methods to detect label drift (or prior probability shift), where the label distribution $P(Y)$ changes.

When choosing a detector, consider whether you have access to ground-truth labels ($y$) or only input features ($X$).

The availability of data and the type of drift you want to detect determine the operational mode of the detector:

• Concept Drift Detectors: In frouros, these are currently implemented exclusively in streaming mode. This means the detector is updated with a single sample at a time.
• Data Drift Detectors: These are more flexible and can be used in either batch or streaming modes. They can also be categorized as univariate (monitoring single features) or multivariate (monitoring the distribution of all features simultaneously).