DOCSEARCHbyAlgolia
Submit Docs

formulaic

repository·main·Indexed 19 days ago

https://github.com/matthewwardrop/formulaic

A high-performance Python library for implementing Wilkinson formulas to convert dataframes into model matrices for statistical modeling. It supports pandas, polars, pyarrow, and narwhals-supported dataframes, with output options including numpy arrays, scipy sparse matrices, and pandas DataFrames. Key features include symbolic differentiation of formulas, stateful transformations, and extensible parsing and materialization plugins.

Tokens
21.3K
Snippets
66
Records
87
Agent score
64%

What's inside formulaic

  1. Overview of Formulaic features and capabilities

    main

    Formulaic is a high-performance implementation of Wilkinson formulas for Python designed for statistical modeling.

    Core Capabilities:

    • High Performance: Outperforms R for both dense and sparse model matrices, and significantly outperforms patsy for dense matrices.
    • Extensible: Supports extensible formula parsing and data input/output plugins.
    • Data Compatibility: Works with pandas, polars, pyarrow, and any dataframe representation supported by narwhals.
    • Advanced Math: Supports symbolic differentiation of formulas and model matrices.

  2. Overview of Formulaic

    main

    Formulaic is a high-performance Python implementation of Wilkinson formulas. It is designed to transform dataframes into model matrices suitable for ingestion into modeling frameworks, such as those used for linear regression.

    Key capabilities include:

    • High-performance conversion from dataframes to model matrices.
    • Reusable encoding: Apply the same encoding logic used on one dataset to other datasets.
    • Extensible formula parsing.
    • Extensible data I/O via plugins supporting:
      • Inputs: pandas.DataFrame, pyarrow.Table
      • Outputs: pandas.DataFrame, numpy.ndarray, scipy.sparse.CSCMatrix
    • Symbolic differentiation of formulas and model matrices.

  3. Avoid using model_matrix() in library code

    main

    When integrating Formulaic into a library or package, do not use the high-level model_matrix function. model_matrix is a syntactic sugar wrapper that automatically attempts to include variables from the user's local namespace. In a library context, this can lead to unexpected interactions, as it may treat your library's internal state as the user context, potentially overriding transforms or exposing sensitive internal data.

    Instead, use the lower-level API: Formula(...).get_model_matrix(...).

    # Avoid this in libraries:
# model_matrix("y ~ x", data)

# Use this instead:
from formulaic import Formula
Formula("y ~ x").get_model_matrix(data)

  4. Understand the Formulaic grammar and operators

    main

    Formulaic uses a formula grammar similar to patsy and R. Operators have different precedence levels; higher precedence operators are resolved first. Within the same precedence level, binary operators are left-associative.

    Key Operators

    OperatorArityDescription
    "..."1String literal.
    [0-9]+\.[0-9]+1Numerical literal.
    `...`1Quotes fieldnames containing special characters (e.g., `my|special$column!`).
    {...}1Quotes Python operations (more idiomatic than I(...)), e.g., {`my|col`**2}.
    <function>(...)1Python transform on a column (e.g., my_func(x) is equivalent to {my_func(x)}).
    (...)1Groups operations to override precedence.
    .0Wild-card for the sum of variables in the data not used on the left-hand side.
    ** or ^2Includes all n-th order interactions (e.g., (a+b+c)**2).
    :2Interaction of operands (elementwise product).
    *2Includes additive and interactive effects (e.g., a * b $\rightarrow$ a + b + a:b).
    /2Nested effects (e.g., a / b $\rightarrow$ a + a:b).
    %in%2Inverted nested effects (e.g., b %in% a $\rightarrow$ a / b).
    +2Adds a term to the feature set.
    -2Removes a term from the feature set.
    |2Splits a formula into multiple parts for simultaneous model matrix generation.
    ~1,2Separates target features from input features.
    [ . ~ . ]2[Experimental] Multi-stage formula notation (requires MULTISTAGE feature flag).

  5. Understand Formulaic's formula conventions and behaviors

    main

    When using Formulaic, be aware of these specific behaviors that may differ from vanilla R:

    • Target vs Input: Both sides of the ~ operator use formula grammar. The right-hand side (RHS) attracts an intercept by default. To treat the left-hand side (LHS) as a standard Python operation (like vanilla R), wrap it in a Python operator block: {y1 + y2} ~ a + b.
    • Term Sorting: Formula terms are always sorted first by interaction order and then alphabetically. This ensures that formulas with the same fields always produce the same model matrix.
    • Intercept Handling: Formulaic follows patsy's logic regarding parentheses. b-1 removes the intercept, but (b-1) will include an intercept because the parentheses are resolved first, effectively resulting in 1 + b - 1.
    • Rank Reduction: Formulaic uses an algorithm (from patsy) to reduce the rank of the model matrix to ensure it is structurally full rank, avoiding over-specification issues common in R.

  6. Understand the concept of formulas and model matrices

    main

    In Formulaic, a formula is a concise specification for how raw input data (typically stored in a dataframe) should be prepared for a statistical model.

    Most statistical solvers require two-dimensional numerical matrices (referred to in Formulaic as model matrices) rather than dataframes. A formula automates the translation of a dataframe into these matrices.

    When you provide a formula and a dataframe, Formulaic generates two objects:

    1. A response matrix ($Y$): An $N imes 1$ matrix for the target variable.
    2. A model matrix ($X$): An $N imes K$ matrix containing the intercept and the processed input features.

    This process includes handling:

    • Interactions: Combining features (e.g., a:b).
    • Transformations: Applying functions like scale() to columns.
    • Encoding: Automatically performing one-hot/dummy encoding for categorical variables.
    • Consistency: Remembering transformations (like the mean and variance used in scale()) from the training data so they can be applied identically to new data.
    # Example of a simple formula
y ~ a + b + a:b

# Example of a complex formula with transformations
~ (f1 + f2 + f3) * (x1 + x2 + scale(x3))

  8. Convert dataframes to model matrices with Formulaic

    main

    Formulaic provides high-performance implementation of Wilkinson formulas to convert dataframes into model matrices (design matrices). You can use the Formula class or the model_matrix convenience function.

    Key features include:

    • Support for pandas.DataFrame, polars.DataFrame, pyarrow.Table, and any narwhals-supported dataframe.
    • Support for multiple output formats: pandas.DataFrame, numpy.ndarray, scipy.sparse.CSCMatrix, or narwhals passthrough.
    • Ability to reuse encoding choices from one dataset on others.
    • Symbolic differentiation of formulas.
    import pandas
from formulaic import Formula

df = pandas.DataFrame({
    'y': [0, 1, 2],
    'x': ['A', 'B', 'C'],
    'z': [0.3, 0.1, 0.2],
})

# Using the Formula class
y, X = Formula('y ~ x + z').get_model_matrix(df)

# Using the short-hand model_matrix function
from formulaic import model_matrix
y, X = model_matrix('y ~ x + z', df)

  9. Configure materializers and output formats

    main

    Formulaic selects materialization algorithms based on the input data type (e.g., pandas.DataFrame uses PandasMaterializer). You can influence this behavior in two ways:

    1. Hard-code a materializer: Pass materializer= to .get_model_matrix() if you want to ensure a specific implementation is used.
    2. Set default output formats: You can override default behaviors, such as enabling sparse outputs for high-cardinality categorical factors, by passing output='sparse' to .get_model_matrix() (provided the materializer supports it).
    # Example: forcing sparse output
Formula("y ~ x").get_model_matrix(data, output='sparse')

  10. Migrating from Patsy to Formulaic

    main

    When migrating a project from patsy to formulaic:

    • Review Migration Notes: Check the official migration documentation for differences in API and formula grammar.
    • Handle Edge Cases: While most formulae are identical, small differences exist in edge cases. Expect some friction in highly entrenched use-cases.
    • Rewrite Internals: If your code manually assembles patsy.Term instances, you must rewrite these using Formulaic classes. This is typically a low-effort change that is transparent to end-users.
    • Leverage Flexibility: Formulaic is designed to be more flexible than Patsy, making it easier to customize for specific needs.

  11. Install and run Formulaic benchmarks

    main

    To run the performance benchmarks comparing formulaic against patsy (Python) and model.matrix/sparse.model.matrix (R), follow these steps:

    1. Install formulaic with the benchmark dependencies:
      pip install formulaic[benchmarks]
    2. Clone the repository and run the benchmark script from the root of the checked-out repository:
      python <formulaic_repo>/benchmarks/benchmark.py

    Note: The benchmark will gracefully skip R-based benchmarks if R or the required R dependency Matrix is not installed on your system.

    pip install formulaic[benchmarks]
python <formulaic_repo>/benchmarks/benchmark.py