MOABB (Mother of All BCI Benchmarks)

MOABB 1.6 introduces support for cross-subject transfer learning via moabb.evaluations.CrossSubjectEvaluation. This is achieved using an optional target-calibration slice provided by the CrossSubjectSplitter.

Key Parameters

• calibration_size: A float in [0, 1] representing the fraction of each held-out subject/session pair set aside for adaptation. When calibration_size > 0, each fold is structured as (train, calibration, test).
• calibration_labeled: A boolean. If True, the calibration set includes both X_target_labeled and y_target_labeled.
• cs_mode: An argument using the moabb.evaluations.CrossSubjectMode enum to select predefined modes:
  • TRAIN_ONLY: No adaptation.
  • Unlabeled target adaptation at 20%, 50%, or 100%.
  • Labeled target calibration at 20% or 50%.
  • TRAIN_TRIALWISE: Scores one target trial at a time (prevents exploiting whole-block statistics).

Data Flow

Calibration trials are trial-disjoint from the training and test sets. Pipeline steps can opt-in to receive calibration data via set_fit_request using keys like X_target_unlabeled, X_target_labeled, y_target_labeled, or the subjects array.

# Example conceptual usage of CrossSubjectEvaluation with calibration
from moabb.evaluations import CrossSubjectEvaluation, CrossSubjectSplitter, CrossSubjectMode

evaluation = CrossSubjectEvaluation(
    splitter=CrossSubjectSplitter(calibration_size=0.2, calibration_labeled=True),
    cs_mode=CrossSubjectMode.UNLABELED_20_PERCENT
)

MOABB is organized around four primary abstractions that define the BCI benchmarking workflow:

1. Datasets: Abstract low-level data access. They handle downloading, local storage, and conversion of raw data into MNE raw objects. They support pooling recording sessions per subject or evaluating them separately.
2. Paradigms: Define how raw data is converted into trials ready for decoding. The paradigm determines the necessary preprocessing and how data is structured (e.g., motor imagery vs. ERP).
3. Evaluations: Define how to calculate generalization statistics (like AUC, f-score, or accuracy) from trials. Evaluations specify the splitting strategy, such as within-session, across-session, or across-subject accuracy.
4. Pipelines: Define the sequence of steps required to obtain predictions. Pipelines are typically chains of scikit-learn compatible transformers ending with a scikit-learn compatible estimator.

Additionally, MOABB provides statistical tools, visualization utilities, and a benchmark module that wraps these steps into a single function for quick execution.

This example demonstrates how to set up a basic benchmarking pipeline using MOABB. It involves defining a pipeline (using sklearn), selecting a dataset, defining a paradigm (e.g., frequency bands for imagery), and running a cross-session evaluation.

import moabb
from moabb.datasets import BNCI2014_001
from moabb.evaluations import CrossSessionEvaluation
from moabb.paradigms import LeftRightImagery
from moabb.pipelines.features import LogVariance

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.pipeline import make_pipeline

# Set logging level
moabb.set_log_level("info")

# Define pipelines as a dictionary of sklearn-compatible pipelines
pipelines = {"LogVar+LDA": make_pipeline(LogVariance(), LDA())}

# Initialize dataset and limit subjects for testing
dataset = BNCI2014_001()
dataset.subject_list = dataset.subject_list[:2]

# Define the paradigm (e.g., frequency range for Left/Right imagery)
paradigm = LeftRightImagery(fmin=8, fmax=35)

# Set up and run the evaluation
evaluation = CrossSessionEvaluation(paradigm=paradigm, datasets=[dataset])
results = evaluation.process(pipelines)

print(results.head())