mpi4py Documentation

mpi4py provides Python bindings for the Message Passing Interface (MPI) standard, based on the MPI-2 C++ bindings. It enables parallel computing in Python through several key capabilities:

• Picklable Object Communication: Send and receive any Python object that supports pickling via point-to-point (send/receive) or collective (broadcast, scatter/gather, reductions) operations.
• Buffer Interface Communication: High-performance communication for objects exposing the Python buffer interface (such as NumPy arrays, bytes, str, or array.array) using blocking, nonblocking, or persistent point-to-point and collective operations.
• Process Groups and Topologies: Management of communication domains, including the creation of intra/inter communicators and Cartesian or graph topologies.
• Parallel I/O: Support for parallel file reading and writing using blocking, nonblocking, collective, or noncollective methods, with support for shared file pointers and explicit offsets.
• Dynamic Process Management: Ability to spawn processes, use accept/connect patterns, and perform name publishing/lookup.
• One-Sided Operations: Remote memory access (put, get, accumulate) using both passive target synchronization (start/complete, post/wait) and active target synchronization (lock/unlock).

The mpi4py.MPI module provides the Python interface to the MPI (Message Passing Interface) standard. It is organized into several functional categories of classes and functions:

Classes

• Communication: Core objects for message passing, including Comm (communicators), Intracomm, Intercomm, and various topology communicators (Cartcomm, Graphcomm, etc.), as well as Message objects.
• One-sided operations: Support for Remote Memory Access (RMA) via the Win (window) class.
• Input/Output: Parallel I/O capabilities via the File class.
• Ancillary: Support objects like Datatype, Group, Op, Request, Status, and Info.
• Error handling: Classes for managing errors, such as Errhandler and Exception.

Functions

• Initialization/Finalization: Managing the MPI environment with Init, Init_thread, and Finalize.
• Version inquiry: Checking versions via Get_version and Get_library_version.
• Memory/Address: Low-level memory management (Alloc_mem, Free_mem) and address manipulation (Get_address).
• Timer: High-resolution timing with Wtime and Wtick.
• Dynamic process management: Managing processes via Open_port, Publish_name, etc.
• Error handling: Utilities to inspect errors like Get_error_string and Get_error_class.

Attributes

• The module contains a vast array of constants for MPI error codes (e.g., SUCCESS, ERR_COMM), data types (e.g., BYTE, INT, DOUBLE), operation types (e.g., SUM, MAX), and communicator/window flags.

The MPIPoolExecutor class is a subclass of concurrent.futures.Executor that uses a pool of MPI processes to execute calls asynchronously. This allows you to side-step the Python Global Interpreter Lock (GIL) by performing computations in separate processes.

Key Requirements and Constraints

• Picklability: Only picklable objects can be executed and returned.
• Module Importability: The __main__ module must be importable by worker processes. Consequently, MPIPoolExecutor may not work in interactive interpreters.
• Avoid Recursion: To prevent infinite recursion (where workers attempt to spawn their own pools), always use the if __name__ == '__main__': idiom in your main script.
• Thread Support: The master process uses a separate thread for MPI communication. For optimal performance, your MPI implementation should support MPI.THREAD_MULTIPLE. If it only supports MPI.THREAD_SERIALIZED, mpi4py.futures will use a global lock to serialize calls. If it supports less than MPI.THREAD_SERIALIZED, a RuntimeWarning will be emitted.

import time
from mpi4py.futures import MPIPoolExecutor

with MPIPoolExecutor(max_workers=1) as executor:
    future = executor.submit(time.sleep, 2)
assert future.done()

MPI-2 allows for dynamic process management, enabling the creation of new processes or connecting disjoint groups of processes.

Spawning new processes

Use Intracomm.Spawn on an existing intracommunicator to create a new independent process group. This returns an Intercomm instance (an intercommunicator) to the parent group. The child group can retrieve the matching intercommunicator using the Comm.Get_parent class method.

Client/Server connection

Disjoint groups can connect using a service-based approach:

1. Server: Call Open_port to open a port, Publish_name to publish a service, and Intracomm.Accept to wait for connections. After use, call Unpublish_name and Close_port to release resources.
2. Client: Call Lookup_name to find a service and obtain its port, then call Intracomm.Connect to establish the connection.

Both Accept and Connect return an Intercomm instance. When communication is finished, all participating processes must call Comm.Disconnect.