DOCSEARCHbyAlgolia
Submit Docs

pymer4: Formula-based Modeling in Python

repository·main·Indexed 42 days ago

https://github.com/ejolly/pymer4

A Python library for estimating regression models, multi-level models, and generalized-linear-mixed models (GLMMs) by interfacing with R via rpy2. It brings R's formula-based modeling syntax to the Python scientific ecosystem and is built on top of the polars DataFrame library. Key model constructors include lm() for linear regression, lmer() for multi-level models, glm() for generalized linear models, and glmer() for generalized linear-mixed models.

Tokens
24.2K
Snippets
90
Records
122
Agent score
52%

What's inside pymer4

  1. Overview of Pymer4 for regression and mixed modeling

    main

    Pymer4 is a statistics library designed for estimating regression models, multi-level models, and generalized-linear-mixed models (GLMMs) within the Python ecosystem. It serves as a Python-based alternative to R's lm(), glm(), lmer(), and glmer() functions.

    Key features include:

    • Formula-based modeling: Provides a unified interface similar to R's modeling syntax.
    • R Integration: Uses rpy2 to interface with R, allowing users to leverage R's robust modeling capabilities without leaving Python.
    • Quality-of-life features: Automatically calculates fit statistics, residuals, p-values for mixed models, bootstrapped confidence intervals, and random-effects deviances.
    • Polars Integration: Built on top of the polars DataFrame library for high performance and efficiency.

  2. Explore pymer4.tidystats submodules

    main

    The tidystats module is organized into several submodules that wrap various R libraries and provide specialized utilities:

    • base: Wraps R base functionality.
    • broom: Wraps broom and broom.mixed for tidying model outputs.
    • easystats: Wraps various sub-libraries from the easystats ecosystem.
    • emmeans_lib: Wraps the emmeans library for estimated marginal means.
    • lmerTest: Wraps lme4 and lmerTest for mixed-effects models.
    • multimodel: Contains functions that use 'function overloading' to switch behavior based on whether the input is an lm, glm, lmer, or glmer model.
    • stats: Wraps the R stats library.
    • tibble: Wraps the tibble library.
    • plutils: Utility functions for working with polars dataframes (includes RandomExpr).
    • tables: Functions to generate formatted summary tables using great tables.
    • bridge: A special module for converting between R and Python datatypes, useful for extending pymer4 functionality.

  3. How the .fit() method works

    main

    When you call .fit() on a model, the library executes the following sequence of steps, passing keyword arguments through to the underlying R functions in pymer4.tidystats:

    1. ._initialize(): Creates an R-model object with optional contrasts and weights.
    2. ._get_design(): Saves the model's design-matrix.
    3. ._get_params(): Estimates fixed-effects terms using parameters::model_parameters() (using Satterthwaite degrees-of-freedom for lmer() models).
    4. ._get_fit_stats(): Retrieves quality-of-fit estimates using broom/broom.mixed::glance().
    5. ._get_fits_resids(): Adds columns to .data using broom/broom.mixed::augment(). Predictions ('fitted') are sourced from the model's .predict() method to support type_predict = 'response'.

    Extending Models: To create new model classes, inherit from an existing model and define a new .fit() method that calls super().fit(*args, **kwargs) followed by your custom logic.

  5. Understand the pymer4 Library Architecture

    main

    The library follows a "functional-core, imperative-shell" pattern.

    1. Functional Core: pymer4.tidystats.bridge contains functions and decorators that handle the conversion between R and Python types, enabling the use of various R libraries within Python.
    2. Imperative Shell: The pymer4.models module provides the high-level API through four core classes:
      • pymer4.models.lm.lm: Linear models with bootstrapping support.
      • pymer4.models.lmer.lmer: Linear mixed-effects models with bootstrapping and random-effects support.
      • pymer4.models.glm.glm: Generalized linear models (extends lm) with support for family and link functions.
      • pymer4.models.glmer.glmer: Generalized linear mixed-effects models (extends lmer) with support for family and link functions.

    All models inherit from pymer4.models.base.model, which implements shared core functionality.

  6. Use scikit-learn compatible estimators with skmer()

    main

    The pymer4.models.skmer module provides estimators that adhere to the scikit-learn API. This makes them compatible with standard scikit-learn tools for model validation, estimation, and prediction.

    After initializing a model with a formula, use the .fit(X, y) and .predict(X) methods. Both methods expect X (features) and y (observations/labels) to be provided as numpy arrays.

    Important for Multi-level Models:

    • Currently, multi-level models support only one random-effect term.
    • When calling .fit(), the feature matrix X must include an extra column at the end containing the values for the random-effect term.
    • When calling .predict(), X can either include or exclude this extra column. If the column is present, random effects are used for predictions; if absent, predictions are made using fixed effects only.
    from pymer4.models import skmer

# Initialize with a formula
model = skmer("y ~ x1 + x2")

# Fit using numpy arrays
model.fit(X_train, y_train)

# Predict using numpy arrays
preds = model.predict(X_test)

  7. Install pymer4 via conda

    main

    As of version 0.9.0, pymer4 is primarily installed via conda using the ejolly channel. For older versions (0.7.1+), you can use the following command to include necessary dependencies from conda-forge and defaults.

    conda install -c ejolly -c defaults -c conda-forge pymer4

  8. Install pymer4 using Pixi

    main

    Pixi is a modern package manager that uses a pyproject.toml file to manage isolated environments. This method allows you to mix conda and pip packages effectively.

    1. Initialize a project with the required channels:
      pixi init --format pyproject --channel ejolly --channel conda-forge myproject
    2. Add pymer4:
      pixi add pymer4
    3. Add additional dependencies (e.g., for Jupyter or plotting):
      pixi add seaborn ipykernel
    4. Add pip-only packages:
      pixi add --pypi polars
    pixi init --format pyproject --channel ejolly --channel conda-forge myproject
pixi add pymer4

  9. Access example data with get_resource_path

    main

    If you want to run the built-in examples provided in the documentation, use get_resource_path from pymer4.utils to locate the sample CSV files included with the package.

    import os
import pandas as pd
from pymer4.utils import get_resource_path

# Load included example data
df = pd.read_csv(os.path.join(get_resource_path(), 'sample_data.csv'))

  10. Import and use pymer4 models

    main

    The pymer4.models module provides four primary model types that share a consistent API. You can import them directly from pymer4.models to perform linear regression, generalized linear modeling, or multi-level modeling.

    Available models:

    • lm(): Linear regression via ordinary-least-squares (OLS).
    • glm(): Generalized linear models (e.g., logistic regression) via maximum-likelihood-estimate (MLE).
    • lmer(): Linear-mixed / multi-level models.
    • glmer(): Generalized linear-mixed / multi-level models.
    from pymer4.models import lm, glm, lmer, glmer

  11. Migrate from pymer4 versions older than v0.9.X

    main
    If you are upgrading from a version older than v0.9.X, note that the library has moved to a more consistent API and offloads computations to R. A major change is the shift from pandas dataframes to polars dataframes. Python now primarily serves as a wrapper to convert inputs and outputs to/from R functions using the tidystats module.