repository·main·Indexed 23 days ago
https://github.com/skrub-data/skrubA Python library for machine learning with tabular data, such as pandas and polars DataFrames. It provides a scikit-learn-compatible API for data exploration, column-level featurization, and multi-column operations. Key features include TableReport for generating interactive HTML summaries of dataframes and DataOp, which encapsulates complete machine learning pipelines from data loading and wrangling to prediction while preventing data leakage.
skrub is a Python library designed to facilitate machine learning workflows specifically with dataframes. It provides tools to bridge the gap between raw dataframes and machine learning models.
For practical applications, you can explore:
Skrub is a Python library designed to facilitate machine learning with tabular data (such as pandas and polars DataFrames). It provides a scikit-learn-compatible API, allowing it to integrate seamlessly into existing machine learning workflows.
Key capabilities include:
Skrub provides a suite of transformers designed for feature engineering on numeric, datetime, and categorical data. These encoders convert raw features from an input DataFrame into numeric features suitable for direct use in machine learning models.
Feature extraction is organized into several specialized modules:
A skrub DataOp is a complete machine learning pipeline encapsulated in a single object. It covers the entire lifecycle from data loading and wrangling to final prediction. Because it follows the scikit-learn estimator interface, a DataOp can be fitted, tuned, cross-validated, and saved to a file just like a standard scikit-learn estimator.
Key advantages of using DataOp over a standard sklearn.pipeline.Pipeline include:
DataOps can handle multiple inputs, allow the number of rows to change (e.g., through filtering or aggregations), and manage complex joins and feature extractions.TableVectorizer or StandardScaler) and estimators (like RandomForestClassifier) within the single object.When standard transformers like StringEncoder are insufficient, use the @skrub.deferred decorator to define custom Python functions that operate on the data.
apply_func with a deferred function to map values (e.g., converting strings to specific integer ranks).Note: Objects inside deferred functions are regular Python objects.
import skrub
import pandas as pd
data = {
"subject": ["Math", "English", "History", "Science", "Art"],
"grade": ["A", "B", "C", "A", "B"]
}
df = pd.DataFrame(data)
grades = skrub.var("grades", df)
# 1. Custom mapping using apply_func
@skrub.deferred
def encode_ordered(df):
grade_order = {"A": 3, "B": 2, "C": 1}
return df["grade"].map(grade_order)
enc_grades = grades.skb.apply_func(encode_ordered)
# 2. Combining results using a deferred function
@skrub.deferred
def combine(subjects, grades):
subjects["grade"] = grades
return subjects
# Assuming enc_subject was created via StringEncoder
# result = combine(enc_subject, enc_grades).skb.set_name(), you can bypass its actual computation by providing a value for that name in the environment dictionary passed to .skb.eval(). This allows you to override specific intermediate results in the computational graph.When writing code for skrub, adhere to these core principles:
_).Selectors in skrub.selectors can be manipulated using standard operators to create complex selection logic.
-): Use this to exclude specific columns from a selection. For example, s.all() - s.cols('name') selects everything except 'name'.~ or s.inv()): Use the tilde operator ~ or the s.inv() function to select columns that do not match a given selector.import pandas as pd
from skrub import SelectCols
import skrub.selectors as s
df = pd.DataFrame({
"height_mm": [297.0, 420.0],
"width_mm": [210.0, 297.0],
"kind": ["A4", "A3"],
"ID": [4, 3],
})
# Exclude specific columns
SelectCols(s.all() - s.cols("height_mm", "width_mm")).fit_transform(df)
# Invert a selector using ~
SelectCols(~s.numeric()).fit_transform(df)
# Invert a selector using s.inv()
SelectCols(s.inv(s.numeric())).fit_transform(df)skrub.TableReport class to explore and understand the contents of your DataFrames. This tool provides structured insights into your data to facilitate data exploration.Parallelism behavior depends on how you invoke the search:
make_randomized_search with backend="optuna":joblib is configured to use processes (default), skrub uses joblib for parallelization.joblib is configured to use the "threading" backend, Optuna's built-in thread-based parallelism is used.timeout: If the timeout parameter is used in make_randomized_search, Optuna's built-in thread-based parallelization is always used, regardless of joblib configuration.In skrub, DataOp objects represent computations that have not been executed yet. They are only triggered when you call .skb.eval() or when you create a pipeline using .skb.make_learner() and call methods like fit().
Because DataOp objects are lazy, you cannot use standard Python control flow (like if, for, or with) directly on them. For example, attempting to iterate over orders.columns will fail because orders.columns is itself a DataOp that will produce a list of columns only when evaluated, not a literal list available immediately.
To use Python control flow, you must wrap the logic in a function that is executed only when the data is actually available.
>>> import pandas as pd
>>> import skrub
>>> orders_df = pd.DataFrame({"item": ["pen", "cup"], "price": [1.5, None], "qty": [1, 1]})
>>> orders = skrub.var("orders", orders_df)
>>> for column in orders.columns:
... pass
TypeError: This object is a DataOp that will be evaluated later...Choices in skrub are not limited to scikit-learn hyperparameters. You can use choice objects anywhere you use DataOps, such as within method arguments or deferred function calls. This allows you to choose between different aggregation methods, preprocessing steps, or even entire pipelines.
To turn a choice between different pipelines into a DataOp that can be used in a sequence, use the .as_data_op() method on the choice object.