DOCSEARCHbyAlgolia
Submit Docs

Anomaly Detection Toolkit (ADTK)

repository·develop·Indexed 22 days ago

https://github.com/arundo/adtk

A Python package for unsupervised and rule-based anomaly detection in time series data. ADTK provides a unified API to build models by combining detectors (to identify anomalies), transformers (for feature engineering), and aggregators (to ensemble results). It includes specialized tools for detecting outliers, spikes, level shifts, and seasonal pattern violations, as well as utilities for data validation, visualization, and pipeline construction via pipe classes.

Tokens
14.2K
Snippets
17
Records
87
Agent score
77%

What's inside ADTK

  1. Overview of Anomaly Detection Toolkit (ADTK)

    develop

    Anomaly Detection Toolkit (ADTK) is a Python package designed for unsupervised and rule-based time series anomaly detection.

    Building an effective anomaly detection model in ADTK involves three core components:

    1. Detectors: Algorithms used to identify anomalies.
    2. Transformers: Feature engineering methods used to process time series data.
    3. Aggregators: Ensemble methods used to combine results from multiple detectors.

    ADTK provides these components with unified APIs and includes pipe classes to connect them into a cohesive model. Additionally, the package offers utility functions for processing and visualizing time series data and anomaly events.

  2. Overview of ADTK core components

    develop

    ADTK is a Python package designed for unsupervised and rule-based time series anomaly detection. It uses a unified API to compose models using three main types of components:

    • Detectors: Algorithms used to identify anomalies.
    • Transformers: Feature engineering methods used to process time series data.
    • Aggregators: Ensemble methods used to combine multiple detection results.

    These components can be connected using pipe classes to build complete anomaly detection models. The toolkit also includes utilities for processing and visualizing time series and anomaly events.

  3. Handle Univariate vs. Multivariate time series

    develop

    ADTK handles both univariate and multivariate data:

    • Univariate: Use components from adtk.transformer and adtk.detector designed for single series.
    • Multivariate (Separable): If anomalies can be detected per dimension (e.g., temperature and humidity separately), applying a univariate detector to a pandas DataFrame will automatically apply it to every series in the DataFrame.
    • Multivariate (Intrinsic): If the anomaly depends on the relationship between dimensions (e.g., a hybrid metric like 'heat index'), you must use multivariate transformers and detectors.

  4. Detect Seasonality in time series

    develop

    A seasonal pattern has a fixed, interpretable period (e.g., hourly, daily, weekly).

    To detect anomalies in seasonal data:

    • Use adtk.detector.SeasonalAD.
    • This detector uses adtk.transformer.ClassicSeasonalDecomposition to remove the seasonal pattern and examines the residual series to find periods that do not follow the normal seasonal pattern.

    Warning: Do not use this for cyclic series (where the period length varies, like rotating equipment). Decomposition of cyclic series produces misleading residuals. For cyclic series, consider adtk.detector.AutoregressionAD to capture changes in autoregressive relationships.

  5. Compose models using Detectors, Transformers, Aggregators, and Pipes

    develop

    ADTK models are built by combining four core component types:

    1. adtk.detector (Detector): Scans time series and returns anomalous time points.
    2. adtk.transformer (Transformer): Transforms time series to extract useful features (feature engineering).
    3. adtk.aggregator (Aggregator): An ensemble component that combines different detection results (anomaly lists).
    4. adtk.pipe (Pipe/Pipenet): Connects components into a model.
      • Use adtk.pipe.Pipeline for sequential combinations (e.g., Transformer $\rightarrow$ Detector).
      • Use adtk.pipe.Pipenet for complex, non-sequential combinations.

    Custom Components: If a component is not implemented, you can wrap a custom function using:

    • adtk.detector.CustomizedDetector1D / CustomizedDetectorHD
    • adtk.transformer.CustomizedTransformer1D / CustomizedTransformerHD
    • adtk.aggregator.CustomizedAggregator

  6. Detect Pattern Changes using Rolling Aggregates

    develop

    You can detect shifts in patterns (like volatility) or temporal changes (like frequency of events) using transformers:

    • adtk.transformer.DoubleRollingAggregate: Used to detect shifts in patterns by comparing statistics between two windows. It supports 16 common statistics (e.g., mean, median, standard deviation). For example, using standard deviation can help detect volatility shifts.
    • adtk.transformer.RollingAggregate: Slides a single window and returns a statistic to quantify a temporal pattern. For example, tracking the count of non-zero values in a window can help detect a temporary high frequency of requests.

  7. Detect Outliers in time series

    develop

    An outlier is a data point that is significantly different from others, regardless of its temporal position (time-independent).

    To detect outliers, you can use:

    • Absolute thresholds: Define a normal range manually using adtk.detector.ThresholdAD.
    • Learned normal ranges: Use detectors that learn the normal range from historical data, such as:
      • adtk.detector.QuantileAD
      • adtk.detector.InterQuartileRangeAD
      • adtk.detector.GeneralizedESDTestAD

  8. Understand the ADTK model class hierarchy

    develop

    ADTK organizes its functionality into a hierarchical structure of model classes. Understanding this hierarchy helps you identify whether a component is a Detector (used to find anomalies), a Transformer (used to preprocess or transform data), or an Aggregator (used to combine results).

    Key distinctions in the hierarchy include:

    • Trainable vs. Non-Trainable:
      • _TrainableModel classes (e.g., QuantileAD, PcaAD) require a training step to learn parameters from data.
      • _NonTrainableModel classes (e.g., ThresholdAD, RollingAggregate) operate based on fixed parameters or simple statistical rules without a training phase.
    • Univariate vs. Multivariate:
      • Univariate models operate on a single time series.
      • Multivariate models operate on multiple time series simultaneously.
    • Detectors, Transformers, and Aggregators:
      • Detectors identify anomalies in the data.
      • Transformers modify the data (e.g., scaling, decomposition, or projection).
      • Aggregators combine multiple anomaly detection results (e.g., using AndAggregator or OrAggregator).
    _Model
    |-- _NonTrainableModel
    |       |-- _NonTrainableUnivariateModel
    |       |       |-- _NonTrainableUnivariateDetector
    |       |       |       └-- ThresholdAD
    |       |       |       └-- _NonTrainableUnivariateTransformer
    |       |       |               |-- RollingAggregate
    |       |       |               |-- DoubleRollingAggregate
    |       |       |               |-- Retrospect
    |       |       |               └-- StandardScale
    |       |       |
    |       |       └-- _NonTrainableMultivariateModel
    |       |               └-- _NonTrainableMultivariateTransformer
    |       |                       └-- SumAll
    |       |
    |       └-- _NonTrainableMultivariateModel
    |               └-- _NonTrainableMultivariateTransformer
    |                       └-- SumAll
    |
    |-- _TrainableModel
    |       |-- _TrainableUnivariateModel
    |       |       |-- _TrainableUnivariateDetector
    |       |       |       |-- QuantileAD
    |       |       |       |-- InterQuartileRangeAD
    |       |       |       |-- GeneralizedESDTestAD
    |       |       |       |-- PersistAD
    |       |       |       |-- LevelShiftAD
    |       |       |       |-- VolatilityShiftAD
    |       |       |       |-- SeasonalAD
    |       |       |       |-- AutoregressionAD
    |       |       |       └-- CustomizedDetector1D
    |       |       |
    |       |       └-- _TrainableUnivariateTransformer
    |       |               |-- ClassicSeasonalDecomposition
    |       |               └-- CustomizedTransformer1D
    |       |
    |       └-- _TrainableMultivariateModel
    |               |-- _TrainableMultivariateDetector
    |       |       |       |-- MinClusterDetector
    |       |       |       |-- OutlierDetector
    |       |       |       |-- RegressionAD
    |       |       |       |-- PcaAD
    |       |       |       └-- CustomizedDetectorHD
    |       |       |
    |       |       └-- _TrainableMultivariateTransformer
    |       |               |-- RegressionResidual
    |       |               |-- PcaProjection
    |       |               |-- PcaReconstruction
    |       |               |-- PcaReconstructionError
    |       |               └-- CustomizedTransformerHD
    |
    └-- _Aggregator
            |-- AndAggregator
            |-- OrAggregator
            └-- CustomizedAggregator

  9. Detect Spikes and Level Shifts

    develop

    Unlike outliers, spikes and level shifts are time-dependent anomalies where a value's normality depends on its recent past.

    • Spike: An abrupt, temporary increase or decrease in value.
    • Level Shift: An abrupt, permanent change in the series value.

    ADTK provides specific detectors for these:

    • adtk.detector.PersistAD for spikes.
    • adtk.detector.LevelShiftAD for level shifts.

    Both are implemented using the adtk.transformer.DoubleRollingAggregate transformer, which compares statistics (like mean or median) across two side-by-side time windows.

  10. Understand the ADTK modeling approach

    develop

    ADTK is designed for unsupervised/rule-based anomaly detection in time series. It does not require labeled historical data, making it suitable for real-world scenarios where anomalies are rare or undocumented.

    If your task requires supervised learning (training on specific normal/anomalous labels), ADTK is not the appropriate tool; you should use alternative machine learning libraries.

  11. Importing ADTK modules

    develop

    As of version 0.6.0, ADTK has a formalized module structure. Second-order sub-modules are private, and users should only import from the following first-order modules:

    • adtk.detector: For anomaly detection models.
    • adtk.transformer: For data transformation models.
    • adtk.aggregator: For data aggregation.
    • adtk.pipe: For pipeline construction.
    • adtk.data: For data manipulation and preparation.
    • adtk.metrics: For evaluation metrics.
    • adtk.visualization: For plotting and visualization.

  12. Launch an interactive demo notebook in Binder

    develop

    You can explore the Anomaly Detection Toolkit (ADTK) capabilities by launching an interactive demo notebook via Binder. This allows you to run ADTK examples in a browser-based Jupyter environment without local installation.

    https://mybinder.org/v2/gh/arundo/adtk/master?filepath=docs%2Fnotebooks%2Fdemo.ipynb