DOCSEARCHbyAlgolia
Submit Docs

TSCV (Time Series Cross-Validation)

repository·master·Indexed 42 days ago

https://github.com/wenjiez/tscv

A scikit-learn extension for time series cross-validation that prevents information leakage by introducing gaps between training and testing sets. It provides specialized cross-validator classes including GapKFold, GapLeavePOut, GapRollForward, GapWalkForward, and GapCrossValidator, as well as the gap_train_test_split utility function.

Tokens
3.4K
Snippets
10
Records
20
Agent score
53%

What's inside TSCV

  1. Core components of the TSCV package

    master

    The tscv package provides tools to introduce gaps in both single train-test splits and multi-fold cross-validation. It consists of three main classes and one utility function:

    Cross-Validation Classes

    These classes can be passed directly to Scikit-Learn functions like cross_validate, cross_val_score, or cross_val_predict via the cv argument:

    • GapLeavePOut
    • GapKFold
    • GapRollForward

    Utility Function

    • gap_train_test_split: A function designed to complement Scikit-Learn's train_test_split by allowing the specification of a gap between the training and testing sets.

  2. What is TSCV and why use it for time series?

    master

    TSCV (Time Series Cross-Validation) is a Scikit-Learn extension designed to handle temporal dependence in time series data.

    Standard cross-validation methods like K-Fold or simple train-test splits assume that data points are independent and identically distributed (i.i.d.). In time series, however, data points are often temporally dependent (e.g., $y_t$ might depend on $x_{t-1}$). Using standard splits can lead to data leakage, where the training set contains information that allows the model to 'cheat' by predicting the test set based on temporal proximity rather than true underlying patterns.

    TSCV mitigates this by introducing gaps between the training set and the test set. These gaps act as a buffer to reduce the influence of temporal dependence, helping to ensure the test set is more independent of the training set.

  3. Integrate GapKFold with scikit-learn

    master
    While you can call .split() directly to verify your configuration and indices, the intended way to use GapKFold in a machine learning pipeline is to pass the GapKFold instance as an argument to a scikit-learn cross-validator (such as sklearn.model_selection.cross_val_score).

  4. Integrate TSCV splitters with Scikit-Learn

    master

    The TSCV classes (GapLeavePOut, GapKFold, and GapRollForward) are designed to be used as splitters within Scikit-Learn's cross-validation functions. Instead of calling them directly, you pass an instance of a TSCV splitter to Scikit-Learn functions like cross_val_score or cross_validate via the cv parameter.

    Note: While the example below uses the Iris dataset, TSCV splitters are specifically designed for time series data where temporal gaps are required between training and testing sets.

    import numpy as np
from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import cross_val_score
from tscv import GapKFold

# 1. Prepare data and model
iris = datasets.load_iris()
clf = svm.SVC(kernel='linear', C=1)

# 2. Instantiate a TSCV splitter
# n_splits: number of folds
# gap_before: number of samples to skip before the test set
# gap_after: number of samples to skip after the test set
cv = GapKFold(n_splits=5, gap_before=5, gap_after=5)

# 3. Pass the splitter to a Scikit-Learn cross-validator
scores = cross_val_score(clf, iris.data, iris.target, cv=cv)

print(scores)

  7. Use GapLeavePOut for time series cross-validation

    master

    The GapLeavePOut class implements a Leave-P-Out cross-validation strategy specifically designed for time series. Unlike standard Leave-P-Out where test points can be non-contiguous, GapLeavePOut ensures the test set is a contiguous block of p points. This allows you to specify gap_before and gap_after parameters to control the buffer between the training and testing sets, which is useful for minimizing data loss or preventing temporal leakage.

    To use it, instantiate GapLeavePOut with your desired parameters and call the .split(indices) method to obtain training and testing indices. In a production machine learning workflow, you should pass the GapLeavePOut instance directly to a scikit-learn cross-validator.

    from tscv import GapLeavePOut

# p: number of points in the test set
# gap_before: number of points to skip before the test set
# gap_after: number of points to skip after the test set
cv = GapLeavePOut(p=3, gap_before=1, gap_after=2)

# split returns (train_indices, test_indices)
for train, test in cv.split(range(7)):
    print("train:", train, "test:", test)

  8. Use TSCV cross-validators with scikit-learn

    master

    TSCV provides three cross-validator classes designed to mitigate temporal dependence and prevent information leakage by introducing gaps between training and test sets. These classes are compatible with scikit-learn functions like cross_validate, cross_val_score, and cross_val_predict via the cv argument.

    Available cross-validator classes:

    • GapLeavePOut
    • GapKFold
    • GapRollForward
    import numpy as np
from sklearn import datasets
from sklearn import svm
from sklearn.model_selection import cross_val_score
from tscv import GapKFold

iris = datasets.load_iris()
clf = svm.SVC(kernel='linear', C=1)

# use GapKFold as the cross-validator
# gap_before and gap_after define the number of samples to skip between train and test sets
cv = GapKFold(n_splits=5, gap_before=5, gap_after=5)
scores = cross_val_score(clf, iris.data, iris.target, cv=cv)

  10. Use GapKFold for time series cross-validation

    master

    In time series analysis, standard K-Fold cross-validation is inappropriate because it shuffles data, breaking temporal dependencies. GapKFold solves this by dividing the sample into $K$ folds and adding 'gaps' on both sides of the test fold. These gaps are taken from neighboring training folds, ensuring that the training and test sets are separated by a buffer to prevent data leakage.

    To use it, instantiate GapKFold with your desired number of splits and gap sizes, then use the .split() method to generate training and test indices.

    from tscv import GapKFold

# n_splits: number of folds
# gap_before: number of samples to skip before the test set
# gap_after: number of samples to skip after the test set
cv = GapKFold(n_splits=5, gap_before=2, gap_after=1)

# split() takes the range of your data indices
for train, test in cv.split(range(10)):
    print("train:", train, "test:", test)

  12. Use GapRollForward for Walk Forward Cross-Validation

    master

    The GapRollForward class implements the 'Walk Forward' (or Rolling Forward) cross-validation method specifically for time series data. Unlike standard K-Fold methods, GapRollForward ensures that the training set always precedes the test set, simulating real-world scenarios where past data is used to predict future data.

    It also allows for a gap_size between the training and test sets to prevent data leakage.

    To use it, instantiate GapRollForward with your desired constraints and call the .split() method to obtain training and testing indices. In a production scikit-learn workflow, you pass the GapRollForward instance directly to a scikit-learn cross-validator.

    from tscv import GapRollForward

# Initialize with training size, gap size, and test size constraints
cv = GapRollForward(min_train_size=3, gap_size=1, max_test_size=2)

# Split a dataset of 10 indices
for train, test in cv.split(range(10)):
    print("train:", train, "test:", test)

# Output:
# train: [0 1 2] test: [4 5]
# train: [0 1 2 3 4] test: [6 7]
# train: [0 1 2 3 4 5 6] test: [8 9]