OptBinning Documentation

You can compute the optimal binning of a variable from a large dataset in a distributed fashion using PySpark's mapPartitions and treeReduce methods combined with OptimalBinningSketch.

To implement this MapReduce pattern:

1. Partition the data: Use df.repartition(n) to split your dataset into multiple partitions.
2. Map Phase: Use mapPartitions to create an OptimalBinningSketch instance for each partition. Inside the partition function, convert the partition to a Pandas DataFrame, initialize OptimalBinningSketch(eps=...), and call .add(x, y) where x is the feature and y is the target.
3. Reduce Phase: Use treeReduce with a merge function that calls optbsketch.merge(other_optbsketch) to aggregate the sketches from all partitions into a single global sketch.

Note: Ensure spark.sql.execution.arrow.enabled is set to true for efficient data transfer between Spark and Pandas.

from pyspark.sql import SparkSession
import pandas as pd
from optbinning import OptimalBinningSketch

# Setup Spark
spark = SparkSession.builder.getOrCreate()
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

# Load and repartition data
df = spark.read.csv("data/kaggle/HomeCreditDefaultRisk/application_train.csv", sep=",", header=True, inferSchema=True)
n_partitions = 4
df = df.repartition(n_partitions)

# Define variables
variable = "EXT_SOURCE_3"
target = "TARGET"
columns = [variable, target]

# Map function: creates a sketch per partition
def add(partition):
    df_pandas = pd.DataFrame.from_records(partition, columns=columns)
    x = df_pandas[variable]
    y = df_pandas[target]
    optbsketch = OptimalBinningSketch(eps=0.001)
    optbsketch.add(x, y)
    return [optbsketch]

# Merge function: aggregates sketches
def merge(optbsketch, other_optbsketch):
    optbsketch.merge(other_optbsketch)
    return optbsketch

# Execute distributed computation
optbsketch = df.select(columns).rdd.mapPartitions(lambda partition: add(partition)).treeReduce(merge)

The OptimalBinningSketch supports a map-reduce pattern for processing large datasets distributed across different files or nodes. You can create multiple sketch objects for different data partitions and then merge them into a single global sketch using the .merge(other_optbsketch) method.

This approach is ideal for:

• Distributed infrastructure: Processing data without centralizing it.
• Large datasets: Handling data that exceeds memory capacity.
• Federated learning: Merging sketches sent from different devices to a central cloud.

from functools import reduce

def add_partition(filepath):
    df = pd.read_parquet(path=filepath, columns=[variable, target])
    x = df[variable].values
    y = df[target].values
    optbsketch = OptimalBinningSketch(name=variable, dtype="numerical", sketch="gk", min_bin_size=0.05)
    optbsketch.add(x, y)
    return optbsketch

def merge_sketches(optbsketch, other_optbsketch):
    optbsketch.merge(other_optbsketch)
    return optbsketch

# Map-reduce pattern
filepaths = ["data/df1.parquet.gzip", "data/df2.parquet.gzip", "data/df3.parquet.gzip"]
optbsketch = reduce(merge_sketches, map(add_partition, filepaths))

optbsketch.solve()

You can force the event rate curve to follow a specific trend using the monotonic_trend parameter. This is useful when business constraints require a specific relationship between the feature and the target.

• auto (default): Uses a machine-learning-based classifier (e.g., GradientBoostingClassifier) to choose the trend (ascending, descending, peak, or valley) that maximizes Information Value.
• Specific trends: You can explicitly set trends like convex or concave.

To use auto, you must provide an estimator (e.g., from sklearn) to the OptimalPWBinning constructor.

from sklearn.ensemble import GradientBoostingClassifier

# Force a convex trend using a Gradient Boosting estimator
optb = OptimalPWBinning(name=variable, 
                        estimator=GradientBoostingClassifier(),
                        monotonic_trend="convex")
optb.fit(x, y)