narwhals

Narwhals is an extremely lightweight and extensible compatibility layer designed to allow developers to write dataframe-agnostic code. It provides a unified interface that works across multiple dataframe libraries without requiring them as dependencies.

Supported Libraries

• Full API support (Eager): cuDF, Modin, pandas, Polars, PyArrow.
• Lazy-only support: Daft, Dask, DuckDB, Ibis, PySpark, SQLFrame.

Key Features

• Polars-based API: Uses a subset of the Polars API, so no new syntax needs to be learned.
• Zero dependencies: Narwhals only uses the objects passed to it by the user, ensuring your project remains lightweight.
• Lazy and Eager separation: Provides separate APIs for lazy and eager execution using expressions.
• Full static typing: Leverages narwhals.typing to provide IDE support and type safety.
• Negligible overhead: Designed to be highly performant with minimal impact on execution speed.

Narwhals is designed to accommodate both pandas users who rely on Index power and those who prefer to ignore it. It follows two core principles to ensure predictable behavior:

1. Index Preservation: Narwhals preserves your original index for most common dataframe operations. This prevents unexpected index resets. Note: Narwhals will not preserve the original index for DataFrame.group_by operations, as overlapping index and column names would cause errors.

2. Positional Alignment (Left-hand Rule): Unlike pandas, which performs automatic index alignment (which can lead to unexpected results when sorting), Narwhals preserves the index of the left-hand-side argument. All other data is inserted positionally, similar to Polars' behavior. This avoids the

import narwhals as nw
import pandas as pd

# Example of Index Preservation
def my_func(df: nw.typing.IntoFrameT) -> nw.typing.IntoFrameT:
    df = nw.from_native(df)
    df = df.with_columns(a_plus_one=nw.col("a") + 1)
    return nw.to_native(df)

df = pd.DataFrame({"a": [2, 1, 3], "b": [3, 5, -3]}, index=[7, 8, 9])
print(my_func(df))  # The result still has the original index [7, 8, 9]

Narwhals operations generally preserve null values. For arithmetic operations like nw.col('a') * 2, non-null values are processed and null values remain null.

However, for boolean comparison operations (e.g., nw.col('a') > 0), null preservation behavior depends on the underlying backend:

• Polars and PyArrow: Null values are preserved.
• pandas: Behavior depends on the dtype backend:
  • PyArrow dtypes or pandas nullable dtypes: Null values are preserved.
  • Classic NumPy dtypes: Null values are typically filled in with False because these dtypes do not support nulls.

import narwhals as nw
from narwhals.typing import IntoFrameT

data = {"a": [1.4, None, 4.2]}

def multiplication(df: IntoFrameT) -> IntoFrameT:
    return nw.from_native(df).with_columns((nw.col("a") * 2).alias("a*2")).to_native()

# For comparison operations:
def comparison(df: FrameT) -> FrameT:
    return nw.from_native(df).with_columns((nw.col("a") > 2).alias("a>2")).to_native()

In Narwhals, an expression is fundamentally a function that maps a DataFrame to a sequence of Series.

For example:

• nw.col('a') is a function that, given a dataframe df, returns the Series 'a' from df.
• nw.col('a') + 1 is a function that takes the Series 'a' and adds 1 to every element.
• nw.col('a', 'b') returns a sequence containing both Series 'a' and 'b'.
• nw.sum_horizontal('a', 'b') takes multiple columns as input and returns a single Series representing their horizontal sum.

An expression does not produce a value by itself; it must be executed within a DataFrame context:

• DataFrame.select: Produces a new DataFrame containing only the results of the expression(s).
• DataFrame.with_columns: Produces a new DataFrame containing the original columns plus the results of the expression(s).
• DataFrame.filter: Evaluates the expression (which must return a single boolean Series) and keeps only the rows where the result is True.

narwhals.LazyFrame is an abstraction used to represent a sequence of data transformations that are not executed immediately. Instead of performing computations on the data, a LazyFrame builds a logical plan of operations. This allows for optimizations and deferred execution, which is common in high-performance data processing engines.

Key capabilities of LazyFrame include:

• Data Transformation: Operations like filter, select, group_by, join, sort, and with_columns define the transformation pipeline.
• Reshaping: Support for unpivot, explode, and drop to change the data structure.
• Execution: The pipeline is eventually executed using methods like collect() (to bring data into memory) or sink_parquet() (to write directly to a file).
• Interoperability: Use to_native() to convert the lazy representation back into the underlying engine's native format (e.g., a Polars or Pandas DataFrame).