FireDucks Documentation

The Import Hook allows you to use FireDucks without modifying your source code. It automatically replaces import pandas statements with FireDucks throughout your execution environment. This is ideal for existing programs or when using external libraries (like matplotlib) that might internally use pandas.

For Python scripts: Run your script using the -m option with the fireducks.pandas module.

For IPython/Jupyter Notebooks: Use the %load_ext magic command to activate the hook before importing pandas.

# For scripts
python3 -m fireducks.pandas your_script.py

# For Jupyter/IPython
%load_ext fireducks.pandas
import pandas as pd

Use the %%fireducks.profile magic command in a Jupyter cell to profile the execution of your code. This is useful for observing the performance benefits of FireDucks' optimizations like Common Sub-expression Elimination, Dead Code Elimination, and Pushdown operations.

%%fireducks.profile
# Your pandas code here
df.groupby("A").sum()

To use FireDucks as a drop-in replacement for pandas in a Jupyter environment, load the fireducks.pandas extension. This allows you to use standard import pandas as pd syntax while benefiting from FireDucks' lazy execution and optimizations.

%load_ext fireducks.pandas
import numpy as np
import pandas as pd

If you prefer to be explicit, you can use the fireducks.pandas module directly. To switch an existing pandas-based program to FireDucks, replace your import pandas statement with import fireducks.pandas.

# Replace this:
# import pandas as pd

# With this:
import fireducks.pandas as pd

FireDucks is available for Linux (manylinux) on the x86_64 architecture. You can install it using pip.

Requirements:

• Python version: 3.9 to 3.13 (Note: Python 3.8 is no longer supported as of FireDucks 1.1.0 due to pyarrow 18.0.0 dependency).

If you use a Python version outside this range, you may encounter the error: "No matching distribution found for fireducks".

pip install fireducks

To achieve maximum performance benefits from the FireDucks JIT compiler, use method chaining for your queries.

When you write a query as a single chained expression, the compiler can optimize the entire data flow because it knows the intermediate results are not needed elsewhere. This allows it to reduce data loading overhead and parallelize the workload efficiently.

# Recommended: Method chaining for maximum JIT optimization
fd.read_parquet("sample_data.parquet").pipe(lambda df: df[df["x"] == 1])["c"].sum()

When working in a Jupyter notebook, avoid saving intermediate results in temporary variables.

If you assign an intermediate result to a variable (e.g., df = fd.read_parquet(...)), the compiler assumes that the variable might be used in a later cell. Consequently, it keeps all intermediate data alive, which prevents the JIT compiler from performing full data-flow optimizations.

Note: This limitation is specific to notebooks. In standard Python scripts, the compiler can inspect the entire file to determine if a variable is used later, allowing for optimization even with intermediate variables.

# Inefficient in notebooks: Intermediate variables prevent optimization
df = fd.read_parquet("sample_data.parquet")
fdf = df[df["x"] == 1]
fdf["c"].sum()

You can install FireDucks using pip. It is recommended to use the -U flag to ensure you are using the latest version.

!pip install -q -U fireducks

Install the FireDucks library using pip. It is recommended to use the -U flag to ensure you are using the latest version.

!pip install -q -U fireducks

#!pip install -q -U fireducks

To use FireDucks alongside Polars and the Linetimer benchmarking tool, install them via pip:

!pip install -q -U fireducks polars linetimer

To use FireDucks, install it along with its required dependencies using pip. This setup is often used in conjunction with pydantic and linetimer for configuration and performance measurement.

!pip install -q -U fireducks pydantic pydantic_settings linetimer