DOCSEARCHbyAlgolia
Submit Docs

Modin Documentation

repository·main·Indexed 27 days ago

https://github.com/modin-project/modin

Modin is a high-performance, drop-in replacement for pandas that scales workflows by utilizing all available CPU cores. It supports larger-than-memory datasets and provides speedups on single machines and clusters using execution engines such as Ray, Dask, and MPI. The documentation covers installation via pip and conda-forge, compute engine configuration, and a layered architecture involving a query planner, executor, and 2D partitioning schema.

Tokens
41.2K
Snippets
89
Records
290
Agent score
94%

What's inside Modin

  1. Overview of Pandas Parsers in Modin

    main

    The modin.core.storage_formats.pandas.parsers module contains the classes and utility functions responsible for data parsing on Modin workers.

    Key components include:

    • PandasParser: The base class for all parser classes using the pandas storage format. It provides common methods used by all child classes.
    • Parser Classes: Specific implementations that implement a parse function. This function parses data for a specific format based on chunk information computed in modin.core.io.

    Parsing Workflow:

    1. The parse function processes specific format data using chunk information.
    2. The resulting DataFrames are split into smaller DataFrames based on the num_splits parameter, data types, or the number of rows/columns in the parsed chunk.
    3. These split frames, along with additional metadata, are returned to the system.

  2. Overview of Modin Component Architecture

    main

    Modin's architecture is composed of several layers that handle the lifecycle of a query:

    1. Pandas API Layer: The user-facing interface that ensures clean input.
    2. Query Compiler: Receives queries from the API layer, optimizes them based on compute kernels and in-memory formats, and sends them to the Core Modin Dataframe.
    3. Core Modin Dataframe: Responsible for data layout, shuffling, partitioning, and serialization. It can execute operations lazily (e.g., transpose).
    4. Partition Manager: Manages partition size/shape (converting between row and column partitions) and handles the shipment of compiled queries to specific partitions.
    5. Partitions: Manage subsets of the Dataframe (row-wise and column-wise) and exploit execution-framework-specific optimizations.
    6. Execution Engine: Performs the actual computation on partitions using task-parallel or data-parallel frameworks.
    7. Storage Format: Defines the in-memory partition type (e.g., pandas objects).

  5. Understand pandas on Dask data transformation flow

    main

    When using modin.pandas.dataframe.DataFrame APIs with pandas on Dask execution, the process follows these steps:

    1. API Layer: Processes the query, determines if the result is a DataFrame or Series, and sanitizes inputs for the compiler.
    2. Query Compiler: The PandasQueryCompiler processes the query to determine how to apply it to data subsets (either cell-wise or axis-wise) backed by the pandas storage format.
    3. Execution Layer: The query is mapped to Core Algebra Operators and executed via the PandasOnDaskDataframe implementation.

  6. Understand pandas on Dask data ingress (reading)

    main

    When performing IO operations (e.g., reading a CSV) via modin.pandas.io with pandas on Dask execution:

    1. The API layer uses PandasOnDaskFactory to access PandasOnDaskIO.
    2. PandasOnDaskIO forwards the query to a dispatcher (e.g., CSVDispatcher._read).
    3. Parameters are checked for compatibility; unsupported parameters default to standard pandas behavior.
    4. The file is split into row chunks, and tasks are deployed to Dask workers using DaskWrapper.deploy.
    5. Each Dask worker uses PandasCSVParser to parse its assigned data chunk.
    6. Results are post-processed, and a new query compiler is returned.

  7. Use generic Ray-based members for storage-agnostic I/O and partitioning

    main

    The modin.core.execution.ray.generic module contains objects that are storage format agnostic but require specific Ray implementations. These components are used to implement parallel I/O operations and serve as a foundation for building storage-format-specific objects within Modin.

    Key components include:

    • modin.core.execution.ray.generic.io.RayIO: Handles parallel I/O operations.
    • modin.core.execution.ray.generic.partitioning.GenericRayDataframePartitionManager: Manages data partitioning for Ray-based execution.

  8. Understand PandasOnPython Data Ingress and Egress

    main

    IO operations in modin.pandas.io using pandas on Python execution are handled via the PandasOnPythonFactory and the PandasOnPythonIO class:

    • Data Ingress (Reading): When reading (e.g., read_csv), PandasOnPythonIO uses the corresponding pandas function. Once reading is complete, a new query compiler is created from the pandas object using PandasOnPythonIO.from_pandas.
    • Data Egress (Writing): When writing (e.g., to_csv), the system converts the query compiler back to a pandas object using BaseQueryCompiler.to_pandas. The pandas object then writes the data to the file using the corresponding pandas function.

  9. Understand Modin Algebra Core Operators

    main

    The modin.core.dataframe.algebra module provides templates for building core operators used by the QueryCompiler. These templates allow you to define how functions are executed across distributed partitions. Note that these templates are currently only supported when using the pandas storage format (i.e., with PandasQueryCompiler).

    Each template implements a register method that takes functions to apply and returns a new function that accepts and returns a QueryCompiler instance. The functions passed to register are executed against converted pandas objects (pandas.DataFrame, pandas.Series, or pandas.DataFrameGroupbyObject).

  11. Understand PandasOnPython Data Transformation flow

    main

    When using modin.pandas.dataframe.DataFrame APIs with pandas on Python execution, the transformation follows this flow:

    1. API Layer: Processes the query, determines if the result is a DataFrame or Series, and sanitizes input for the compiler.
    2. PandasQueryCompiler: Receives the sanitized query and determines how to apply it to data partitions (cell-wise or axis-wise) using the pandas storage format.
    3. Execution: The compiler maps the query to Core Algebra Operators implemented by the PandasOnPythonDataframe.

  12. How PandasOnRay handles data ingress (Reading)

    main

    When using modin.pandas.io for reading data (e.g., CSV files) with pandas on Ray:

    1. Dispatching: The FactoryDispatcher uses PandasOnRayFactory to provide the PandasOnRayIO class.
    2. Preprocessing: For CSVs, the query is forwarded to CSVDispatcher._read, which validates parameters and computes metadata.
    3. Parallel Loading: The file is split into row chunks. These chunks are used to launch remote tasks on Ray workers via RayWrapper.deploy.
    4. Parsing: Each Ray worker uses PandasCSVParser to parse its assigned chunk.
    5. Postprocessing: After remote tasks complete, result postprocessing is performed, and a new query compiler is returned.