polars-ds Documentation

Polars DS (PDS) pipelines are designed to be dataframe-centric and strictly focused on data transformation. Unlike Scikit-learn, PDS pipelines do not include models (e.g., classifiers or regressors) within the pipeline object.

This design choice enforces a separation of concerns:

1. Output Clarity: PDS pipelines output transformed features. This avoids the complexity of managing raw features, transformed features, and model scores in a single object.
2. Efficiency in Hyperparameter Tuning: By keeping models separate, you can tune model hyperparameters without re-running the entire data transformation pipeline every time.
3. API Predictability: PDS pipelines do not implement .predict() or .predict_proba(). This ensures the API remains small and predictable, as the pipeline's sole responsibility is transformation.

While PDS pipelines can be wrapped inside a Scikit-learn Transformer, the reverse is not supported because Scikit-learn lacks native Polars support (e.g., it cannot handle Polars selectors like cs.numeric()).

The polars_ds extension implements smoothing splines by transforming a continuous optimization problem into a discrete linear algebra problem.

Instead of solving a continuous calculus problem involving an integral of squared second derivatives, the algorithm uses the fact that the optimal solution is always a natural cubic spline with knots at the data points $x_i$.

Key components of the mathematical model:

• Objective Function: Minimizes the sum of squared errors (fidelity to data) plus a roughness penalty (the integral of the squared second derivative).
• Discretization: The continuous integral is converted into a quadratic form $\gamma^T R \gamma$ using a symmetric tridiagonal matrix $R$ that represents the interval widths ($h_i = x_{i+1} - x_i$).
• The Penalty Matrix ($K$): A matrix $K = Q R^{-1} Q^T$ is constructed to represent the roughness penalty purely in terms of the spline values $g$.
• The Linear System: The final computation solves the system $(I + \lambda Q R^{-1} Q^T) g = y$, where $I$ is the identity matrix, $\lambda$ is the smoothing parameter, and $y$ is the input data vector. This allows the Rust implementation to find the discrete $g$ coordinates that minimize the continuous objective efficiently.

PDS provides a Pipeline and Blueprint system for creating reproducible data transformation workflows.

1. Initialize a Blueprint: Use Blueprint(df, name=..., target=..., lowercase=True) to define the starting state and the target column. The target column is automatically excluded from certain operations like scaling.
2. Define Transformations: Chain methods such as .filter(), .linear_impute(), .impute(), .scale(), .one_hot_encode(), .woe_encode(), and .target_encode().
3. Append Expressions: Use .append_expr() to add custom Polars expressions (e.g., log transforms, clipping, or missing value flags) to the pipeline.
4. Materialize and Transform: Call .materialize() on the Blueprint to get a Pipeline object, then use .transform(df) to apply the sequence to your data.

import polars as pl
import polars.selectors as cs
from polars_ds.pipeline import Pipeline, Blueprint

bp = (
    Blueprint(df, name = "example", target = "approved", lowercase=True)
    .filter(pl.col("city_category").is_not_null())
    .linear_impute(features = ["var1", "existing_emi"], target = "loan_period") 
    .impute(["existing_emi"], method = "median")
    .append_expr( 
        pl.col("existing_emi").log1p().alias("existing_emi_log1p"),
        pl.col("loan_amount").log1p().alias("loan_amount_log1p"),
        pl.col("loan_amount").clip(lower_bound = 0, upper_bound = 1000).alias("loan_amount_clipped"),
        pl.col("loan_amount").sqrt().alias("loan_amount_sqrt"),
        pl.col("loan_amount").shift(-1).alias("loan_amount_lead_1")
    )
    .scale( 
        cs.numeric().exclude(["var1", "existing_emi_log1p"]), method = "standard"
    )
    .append_expr(
        pl.col("employer_category1").is_null().cast(pl.UInt8).alias("employer_category1_is_missing")
    )
    .one_hot_encode("gender", drop_first=True)
    .woe_encode("city_category")
    .target_encode("employer_category1", min_samples_leaf = 20, smoothing = 10.0)
)

pipe:Pipeline = bp.materialize()
df_transformed = pipe.transform(df)

PDS provides a Blueprint and Pipeline system for building machine learning data transformation pipelines that are compatible with Polars' lazy execution and batch processing.

1. Define a Blueprint with a target column.
2. Chain transformations like linear_impute, impute, scale, one_hot_encode, woe_encode, and target_encode.
3. Call .materialize() to create a Pipeline.
4. Use .transform(df) to apply the pipeline. For large datasets, use .transform(df, return_lazy=True).collect_batches() to process data in chunks.

from polars_ds.pipeline import Pipeline, Blueprint
import polars.selectors as cs

bp = (
    Blueprint(df, name = "example", target = "approved", lowercase=True)
    .linear_impute(features = ["var1", "existing_emi"], target = "loan_period")
    .impute(["existing_emi"], method = "median")
    .scale(cs.numeric().exclude(["var1", "existing_emi_log1p"]), method = "standard")
    .one_hot_encode("gender", drop_first=True)
    .target_encode("employer_category1", min_samples_leaf = 20, smoothing = 10.0)
)

pipe: Pipeline = bp.materialize()
df_transformed = pipe.transform(df)