skore

Skore is a tool designed to turn machine learning development into structured decision-making. It consists of two main components:

• Skore Lib: An open-source Python library that provides structured artifacts (like evaluation reports) and methodological guidance for data science experiments.
• Skore Hub: A collaborative platform for teams to share, compare, and build upon structured experiments.

Skore is a framework designed to structure and store the critical information from your data science experiments. While libraries like pandas, polars, skrub, and scikit-learn provide the building blocks for data transformation and modeling, Skore acts as the glue that ties these pieces together.

Skore works by taking your full data science pipeline as input and producing structured artifacts. These artifacts store the specific information relevant to your use case, reducing boilerplate code and minimizing the overhead of manual documentation and result tracking. This allows you to focus on the impact of your analysis choices and easily retrieve experiment results later.

Skore allows for model validation through a check system:

• Check: A protocol used to define custom checks.
• ChecksSummaryDisplay: A display class for visualizing check results returned by the checks accessor.
• CheckNotApplicable: An exception raised when a specific check cannot be executed on a given report.

A ComparisonReport is used to compare different predictive models. It is generated by evaluate when passed a list or dictionary of estimators, or via the compare() function when you already have existing reports.

Requirements: To ensure a valid comparison, all reports must have the same test target (the y_test array). skore verifies this by computing a hash of the target arrays. If the targets are functionally equal but have different data types, they will be treated as different.

Reports can have different training data or different testing datasets, allowing for comparisons between a new model and a production model.

from skore import evaluate

# Comparing two estimators
report = evaluate({"model_a": est_a, "model_b": est_b}, X, y)

# Access comparison metrics
comparison_summary = report.metrics.summarize()

The SKD001 check compares training and testing scores across default predictive metrics. A metric votes for overfitting if the gap exceeds an adaptive threshold:

• higher-is-better metrics: train - test >= threshold
• lower-is-better metrics: test - train >= threshold

The threshold is calculated as: max(0.03, 0.10 * |reference|), where reference is the training score (for higher-is-better) or the test score (for lower-is-better). A floor of 0.03 is used to prevent the threshold from vanishing near zero. An issue is flagged if a strict majority of metrics vote for overfitting.

The EstimatorReport class allows you to inspect and evaluate a scikit-learn estimator interactively. It organizes its functionality into several specialized accessors: data, metrics, inspection, and checks.

Key methods include:

• help(): Provides guidance on using the report.
• get_predictions(): Retrieves the predictions made by the estimator.

Accessors provide specific domains of insight:

• report.data: Insights into the training and testing datasets.
• report.metrics: Statistical performance evaluation.
• report.inspection: Model inspection (e.g., feature importance).
• report.checks: Automated checks for modeling issues like overfitting or underfitting.

from skore import EstimatorReport

# Assuming 'estimator' is a trained scikit-learn model
report = EstimatorReport(estimator)

The CrossValidationReport class is used to perform cross-validation on a scikit-learn estimator and provides an interactive way to inspect and evaluate the results. It exposes its functionalities through several specialized accessors: data, metrics, inspection, and checks.

from skore import CrossValidationReport

# Usage involves initializing the report and using accessors to inspect results
report = CrossValidationReport(estimator, X, y, ...)
report.metrics.available()

A CrossValidationReport is returned when using evaluate(..., splitter=N) where N is an integer or a CV splitter.

It acts as a collection of EstimatorReport instances (accessible via the reports_ attribute), where each instance corresponds to a single fold. The CrossValidationReport exposes a similar API to EstimatorReport for metrics and displays, but includes an aggregate parameter to compute results across all splits.

from skore import evaluate

# Returns a CrossValidationReport
report = evaluate(estimator, X, y, splitter=5)

# Metrics are aggregated across folds
avg_accuracy = report.metrics.accuracy(aggregate=True)

# Access individual fold reports
first_fold_report = report.reports_[0]