DOCSEARCHbyAlgolia
Submit Docs

miceforest Documentation

repository·master·Indexed 19 days ago

https://github.com/anothersamwilson/miceforest

A fast, memory-efficient implementation of Multiple Imputation by Chained Equations (MICE) using LightGBM as the backend. Version 6.0.5 supports pandas DataFrames, numpy arrays, and GPU acceleration. The library features the ImputationKernel for managing the MICE process and ImputedData for handling results. It includes utilities for simulating missing data via ampute_data, integration with sklearn Pipelines, and tools for tuning LightGBM parameters and ensuring reproducibility.

Tokens
11.5K
Snippets
37
Records
46
Agent score
64%

What's inside miceforest

  1. Core features of miceforest

    master

    miceforest is designed for high-performance data imputation with the following characteristics:

    Performance

    • Fast: Uses lightgbm as a backend, features efficient mean matching, and supports GPU training.
    • Memory Efficient: Supports in-place imputation to save memory and provides efficient compression for saving/loading kernels.

    Flexibility

    • Data Types: Works with pandas DataFrames and handles categorical data automatically.
    • Integration: Designed to fit into sklearn pipelines.
    • Customization: Allows users to customize almost every aspect of the imputation process.

    Production Readiness

    • Inference: Capable of imputing new, unseen datasets quickly.

  3. Understand the MICE algorithm and its use cases

    master

    Multiple Imputation by Chained Equations (MICE) fills in missing data through an iterative series of predictive models. In each iteration, every specified variable is imputed using the other variables in the dataset. Convergence is typically reached within 5 iterations.

    Common Use Cases

    • Data Leakage: Handling missing values that are associated with a target variable (e.g., a variable collected after a certain event).
    • Funnel Analysis: Estimating characteristics of entities at different stages of a process.
    • Confidence Intervals: Using multiple imputed datasets to build a distribution of predictions, allowing for inference on the uncertainty of imputed values.

  4. Core classes: ImputationKernel and ImputedData

    master

    The miceforest library revolves around two primary classes:

    • ImputationKernel: The central object that holds the raw data and manages the MICE (Multiple Imputation by Chained Equations) process. It stores trained models and predicted values. It can be used to perform multiple imputations, save models, and impute new datasets.
    • ImputedData: The object returned by ImputationKernel.impute_new_data(new_data). It contains both the original new_data and the resulting imputed values.

  5. How the MICE algorithm works in miceforest

    master

    miceforest implements Multiple Imputation by Chained Equations (MICE) using LightGBM. The algorithm 'fills in' missing data through an iterative series of predictive models.

    In each iteration, every specified variable in the dataset is imputed using the other variables in the dataset. This process repeats until convergence is met. While the number of iterations can be adjusted, typically no more than 5 iterations are necessary for convergence.

  6. Ensure reproducibility with global and record-level seeds

    master

    miceforest provides two levels of reproducibility:

    1. Global Reproducibility: Set a random_state during ImputationKernel initialization. This ensures the entire imputation process is reproducible across runs.
    2. Record-Level Reproducibility: Use the random_seed_array parameter in impute_new_data(). By passing an array of seeds corresponding to the rows of your data, you ensure that a specific row will always receive the same imputed values every time it is processed, even if the input dataset changes.
    import numpy as np

# 1. Create a seed array matching the shape of your data
random_seed_array = np.random.randint(0, 9999, size=iris_amp.shape[0], dtype='uint32')

# 2. Impute with the seed array
iris_imputed = kernel.impute_new_data(
    iris_amp,
    random_state=4,
    random_seed_array=random_seed_array
)

# 3. When imputing a subset, use the corresponding seeds from the original array
new_inds = np.random.choice(150, size=15)
new_data = iris_amp.loc[new_inds].reset_index(drop=True)
new_seeds = random_seed_array[new_inds]

new_imputed = kernel.impute_new_data(
    new_data,
    random_state=4,
    random_seed_array=new_seeds
)

# new_imputed rows will match the original iris_imputed rows for these indices

  7. Use Predictive Mean Matching (PMM) for specific data types

    master

    Predictive Mean Matching (PMM) selects imputed values from the original, non-missing data points (candidates) that are closest to the predicted value (bachelors).

    This is controlled by the mean_match_candidates parameter, which defines the number of neighbors ($N$) to consider. A value is then chosen at random from these $N$ candidates.

    When to use PMM: Use PMM if your variables are:

    • *Multimodal
    • *Integer
    • *Skewed

    Using PMM helps ensure that the imputed values maintain a distribution similar to the original data, whereas using raw LightGBM predictions might provide a better 'fit' but result in unrealistic distributions.

    # Example: Using PMM with 5 candidates
kernel_mean_match = mf.ImputationKernel(
    data=ampdat,
    num_datasets=3,
    mean_match_candidates=5,
    random_state=1
)
kernel_mean_match.mice(2)

# Example: Skipping PMM (using raw LightGBM predictions)
kernel_no_mean_match = mf.ImputationKernel(
    data=ampdat,
    num_datasets=3,
    mean_match_candidates=0,
    random_state=1
)
kernel_no_mean_match.mice(2)

  8. Perform single and multiple imputation with ImputationKernel

    master

    To perform imputation, you first prepare your data (e.g., by introducing missing values using mf.ampute_data).

    Single Imputation

    For a single imputed dataset, initialize ImputationKernel with default settings, call .mice(iterations), and retrieve the result with .complete_data().

    Multiple Imputation

    To account for uncertainty, you can create multiple datasets by setting the num_datasets parameter in ImputationKernel. This performs mutually exclusive imputation processes across the datasets. You can then retrieve a specific completed dataset using .complete_data(dataset=index).

    import miceforest as mf
import pandas as pd
from sklearn.datasets import load_iris

# Setup data
iris = pd.concat(load_iris(as_frame=True,return_X_y=True),axis=1)
iris.rename({"target": "species"}, inplace=True, axis=1)
iris['species'] = iris['species'].astype('category')
iris_amp = mf.ampute_data(iris, perc=0.25, random_state=1991)

# Multiple Imputation Example
kernel = mf.ImputationKernel(
  iris_amp,
  num_datasets=4,
  random_state=1
)

kernel.mice(2)

# Get the second imputed dataset (index 2)
completed_dataset = kernel.complete_data(dataset=2)

  9. Install miceforest via pip or conda

    master

    You can install miceforest using either pip or conda (via the conda-forge channel).

    To install the latest development version from GitHub using pip, ensure you have git installed.

    # Using pip
$ pip install miceforest --no-cache-dir

# Using conda
$ conda install -c conda-forge miceforest

# Install from GitHub
$ pip install git+https://github.com/AnotherSamWilson/miceforest.git

  10. Impute data in place to save memory

    master

    To avoid copying the dataset and save memory, set copy_data=False when initializing the ImputationKernel.

    Warning: This modifies the original dataset directly during the mice procedure. Imputed values are stored in the original data, and at the end of the process, missing values are restored as np.NaN.

    To apply the completed values back to the original dataframe without creating a new object, use complete_data(dataset=..., inplace=True).

    # Initialize without copying the original dataframe
kernel_inplace = mf.ImputationKernel(
  iris_amp,
  num_datasets=1,
  copy_data=False,
  random_state=1,
)
kernel_inplace.mice(2)

# Apply completed data directly to the original dataframe
kernel_inplace.complete_data(dataset=0, inplace=True)

  11. Save and load ImputationKernels with pickle or dill

    master

    Kernels can be serialized efficiently. During pickling, working data is converted to Parquet bytes before serialization. Use standard Python libraries like pickle or dill to save and load kernels to/from files.

    import dill

# Saving
with open(filename, "wb") as f:
    dill.dump(kernel, f)

# Loading
with open(filename, "rb") as f:
    kernel_from_pickle = dill.load(f)

  12. Optimize the imputation process speed

    master

    Multiple Imputation can be computationally expensive. Use the following strategies to decrease runtime:

    • data_subset: Decrease this value to search a smaller subset of non-missing datapoints for mean matching instead of the full dataset.
    • Categorical Columns: If categorical columns are slow, set mean_match_strategy="fast". You can also tune bagging_fraction or num_iterations specifically for these columns, or group categories before imputation.
    • mean_match_candidates: Decrease this value to reduce the number of neighbors considered during mean matching. Setting mean_match_candidates=0 skips mean matching entirely and uses raw LightGBM predictions.
    • LightGBM Parameters: For variables with many classes, the number of trees grows significantly. Decrease bagging_fraction, n_estimators, or the total number of trees grown for these specific variables.