pandarallel

repository·master·Indexed 26 days ago

A library to parallelize pandas operations across all available CPUs by replacing standard methods with parallel equivalents, such as .parallel_apply(), .parallel_applymap(), and .parallel_map(). It includes built-in progress bar support for terminal and Notebook environments. The library requires twice the memory of standard pandas and requires self-contained functions for Windows compatibility due to the multiprocessing spawn system.

Tokens: 2K
Snippets: 7
Records: 20
Agent score: 86%

What's inside pandarallel

Parallelize Pandas operations with pandarallel
master
Use pandarallel to parallelize standard Pandas operations across all available CPUs. It provides a drop-in replacement for several Pandas methods by prefixing them with parallel_. It also includes progress bars for both terminal and Notebook environments.
Supported API mappings:
- df.apply(func) $\rightarrow$ df.parallel_apply(func)
- df.applymap(func) $\rightarrow$ df.parallel_applymap(func)
- df.groupby(args).apply(func) $\rightarrow$ df.groupby(args).parallel_apply(func)
- df.groupby(args1).col_name.rolling(args2).apply(func) $\rightarrow$ df.groupby(args1).col_name.rolling(args2).parallel_apply(func)
- df.groupby(args1).col_name.expanding(args2).apply(func) $\rightarrow$ df.groupby(args1).col_name.expanding(args2).parallel_apply(func)
- series.map(func) $\rightarrow$ series.parallel_map(func)
- series.apply(func) $\rightarrow$ series.parallel_apply(func)
- series.rolling(args).apply(func) $\rightarrow$ series.rolling(args).parallel_apply(func)
https://github.com/nalepae/pandarallel/blob/master/docs/docs/index.md
Quickstart with pandarallel
master
To use pandarallel, initialize it once at the start of your script and then replace standard pandas .apply() calls with .parallel_apply().
```
from pandarallel import pandarallel

# Initialize with progress bar enabled
pandarallel.initialize(progress_bar=True)

# Instead of df.apply(func), use:
df.parallel_apply(func)
```
https://github.com/nalepae/pandarallel/blob/master/README.md
Compare pandas, pandarallel, and pyspark for data processing
master
Choosing the right tool depends on your data size and hardware constraints:
- pandas: Best for standard data manipulation. It is easy to use but limited to a single CPU core.
- pandarallel: Best for speeding up Pandas operations on multi-core machines. It uses all available cores but requires twice the memory of standard Pandas. Do not use pandarallel if your data cannot fit into memory with standard pandas.
- pyspark: Best for datasets much larger than your available memory or when you need to distribute computation across a cluster of nodes. It requires a JVM.
https://github.com/nalepae/pandarallel/blob/master/docs/docs/index.md
Install pandarallel via pip
master
Install the pandarallel package using pip. You can use --upgrade to update an existing installation or --user to install it for the current user.
```
pip install pandarallel [--upgrade] [--user]
```
https://github.com/nalepae/pandarallel/blob/master/docs/docs/user_guide.md
Ensure functions are self-contained for Windows compatibility
master
On Windows, due to the multiprocessing spawn system, any function passed to pandarallel must be self-contained. This means the function should not depend on external resources or imports defined in the global scope. To ensure compatibility across all platforms (Linux, macOS, and Windows), perform imports inside the function itself.
```
# ✅ Valid everywhere (Self-contained)
def func(x):
    import math
    return math.sin(x.a**2) + math.sin(x.b**2)
```
https://github.com/nalepae/pandarallel/blob/master/docs/docs/index.md
Initialize pandarallel
master
To use parallelization, first import pandarallel and then call pandarallel.initialize().
initialize() accepts the following optional parameters:
- nb_workers (int): The number of workers used for parallelization. If not set, it defaults to the number of available cores.
- progress_bar (bool): If set to True, displays progress bars. Defaults to False.
- verbose (int): The verbosity level. Defaults to 2.
  0: No logs.
  1: Warning logs only.
  2: All logs.
- use_memory_fs (bool): Controls data transfer between the main process and workers. Using a memory file system (like /dev/shm on Linux) reduces transfer time for large datasets.
  None (default): Uses memory file system if available (if /dev/shm exists and is writable); otherwise, defaults to multiprocessing pipes.
  True: Forces use of memory file system; raises SystemError if not available.
  False: Forces use of multiprocessing pipes.
Note: shm_size_mb is deprecated and should not be used.
https://github.com/nalepae/pandarallel/blob/master/docs/docs/user_guide.md
Initialize pandarallel
master
Before using parallelized pandas operations, you must initialize the pandarallel library using pandarallel.initialize().
```
from pandarallel import pandarallel

pandarallel.initialize()
```
https://github.com/nalepae/pandarallel/blob/master/docs/examples_mac_linux.ipynb
Fixing multiprocessing issues on Windows
master
On Windows, pandarallel uses the spawn multiprocessing system. This requires the function passed to pandarallel to be self-contained. The function must not depend on external resources or imports defined in the global scope; instead, all necessary imports must be performed inside the function itself.
```
# ✅ Valid: Self-contained function for Windows
def func(x):
    import math
    return math.sin(x.a**2) + math.sin(x.b**2)
```
https://github.com/nalepae/pandarallel/blob/master/docs/docs/troubleshooting.md
Optimizing worker count for physical cores
master
If you do not see performance increases when increasing the number of workers, you may be exceeding the number of physical cores on your CPU. Hyperthreading (e.g., an 8-CPU system that is actually 4 cores) does not provide additional physical computation units. To find the number of physical cores to use for optimal performance, use psutil.cpu_count(logical=False).
https://github.com/nalepae/pandarallel/blob/master/docs/docs/troubleshooting.md
Fixing progress bar display in Jupyter Lab
master
If progress bars appear as raw text (e.g., VBox(children=...)) instead of visual bars in Jupyter Lab, you need to install and enable ipywidgets and the JupyterLab manager. You may also need to install nodejs if prompted.
```
pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension
jupyter labextension install @jupyter-widgets/jupyterlab-manager
```
https://github.com/nalepae/pandarallel/blob/master/docs/docs/troubleshooting.md
Initialize pandarallel
master
Call pandarallel.initialize() to set up the parallel processing environment. You can pass progress_bar=True to display progress bars during execution.
```
pandarallel.initialize(progress_bar=True)
```
https://github.com/nalepae/pandarallel/blob/master/README.md
Ensure functions are self-contained on Windows
master
On Windows, pandarallel uses the spawn multiprocessing system. This requires that any function passed to pandarallel methods must be self-contained. You must not depend on external resources or imports defined in the global scope; instead, import necessary modules inside the function itself.
Forbidden (Global import):
```
import math
def func(x):
    return math.sin(x.a**2)
```
Valid (Local import):
```
def func(x):
    import math
    return math.sin(x.a**2)
```
https://github.com/nalepae/pandarallel/blob/master/docs/examples_windows.ipynb