MPICH Documentation

TAU is a toolkit designed for profiling and tracing parallel programs written in C, C++, Fortran, and other languages.

Key features include:

• Instrumentation Support: Supports dynamic (library-based), compiler, and source-level instrumentation.
• Broad Parallelism Support: Unlike MPE, TAU is not limited to MPI; it supports CUDA, OpenMP, and regular pthreads.
• General Purpose: Geared towards parallel programming in general rather than just MPI code.

MPICH is a portable implementation of the Message Passing Interface (MPI) standard. It provides full support for all versions of the MPI standard, including:

• MPI-1
• MPI-2
• MPI-3
• MPI-4

PMI version 2 introduces several improvements over version 1:

• Job Information: Improved handling of Job attributes and node information.
• Connection Information: Uses a Key-Value Store (KVS) space to share connection information. This avoids forcing the MPI implementation to read and broadcast the full KVS space during MPI_Comm_spawn, MPI_Comm_connect, MPI_Comm_accept, or MPI_Join operations.
• Simplified KVS: The KVS API is simplified to support only a single KVS space per parallel Job.
• Character Set: Expanded from ASCII to UTF-8.
• ABI Stability: Various sizes have been fixed as part of an Application Binary Interface (ABI) to facilitate the use of dynamically loadable libraries. This allows an executable to load a PMI version compatible with the running process management system without requiring identical wire protocols.

DTPools is a library for MPICH designed to build and manage sets of MPI datatype instances for testing point-to-point, collective, and RMA communications (e.g., MPI_Send, MPI_Recv, MPI_Bcast).

It works by defining "pools" of datatypes. Each pool has a specific signature (a combination of a basic type and a count). Within a pool, multiple datatype instances with different internal layouts can exist. This allows developers to randomly pick pairs of datatypes from the same pool or from different pools (provided they share the same basic type) to test communication robustness.

MPE is a set of postmortem profiling tools for MPI programs. It consists of two main components:

1. libmpe: An instrumentation library. You can use it by linking your program against it (link-time instrumentation) or by manually inserting logging calls into your source code (source-code instrumentation) for higher detail.
2. Jumpshot: A graphical visualizer used to view the profiling data.

While often used with MPICH, MPE is implementation-agnostic and works with OpenMPI, LAM/MPI, and commercial implementations like IBM's and Cray's.

DAME is an efficient internal representation of data movements defined by MPI datatypes. Instead of executing the user's MPI datatype definition directly, MPICH uses DAME to represent these moves as an optimized "program" that can be executed by a high-performance interpreter or compiled into native object code at runtime.

Key characteristics include:

• Optimization: DAME can transform a user's datatype (e.g., a struct) into a more efficient representation (e.g., a contiguous block).
• Partial Moves: It is designed to handle "partial messages" by providing a compact way to describe the state of a move, allowing the engine to stop and resume from the last position.
• Hardware Awareness: The engine can leverage hardware scatter/gather support and optimize move instructions based on data alignment and size (e.g., using different instructions for short vs. long block moves).

Persistent collectives are an API (approving for MPI 4.0) designed for scenarios where a non-blocking collective with the same send/receive buffers is called repeatedly.

By using this API, the MPI runtime is notified that the collective will be reused with the same arguments, allowing it to perform one-time optimizations (like algorithm selection or schedule generation) that are then amortized over multiple calls.

Currently, MPICH supports persistent collectives for the following operations:

• BCAST
• REDUCE
• ALLREDUCE
• ALLTOALL
• ALLGATHER
• ALLGATHER

When working with MPI, you must use the language-specific constants for datatypes. Using a C datatype in a Fortran program (or vice versa) is an error.

• In C: Datatypes are of type MPI_Datatype.
• In Fortran: Datatypes are of type INTEGER.
• In Fortran 08: Datatypes are of type Type(MPI_Datatype).

Example: Do not use MPI_INT for a Fortran INTEGER type.

In addition to the regular connection, a client can open a second "out-of-band" connection to receive asynchronous signals (like SIGSTOP) from the server.

1. Initialization: The client sends: cmd=init_oob signal=SIGSTOP pmi_version=... (or signal=NONE to disable signaling).
2. Server Communication: The server initiates messages on this socket using cmd=checkpoint ... or cmd=abort ....
3. Client Handling: If signal=NONE is used, the client must continuously monitor the socket using SIGIO or a dedicated blocking thread.

# Client OoB initialization
cmd=init_oob signal=SIGSTOP pmi_version=2

For developers working with the ADI3 (Abstract Device Interface) implementation, specifically the CH3 implementation, certain headers are scoped to specific layers:

CH3 Implementation Layer

src/mpid/ch3/include/mpidimpl.h is intended to be included only within the ADI3 implementation. It aggregates:

• mpichconf.h (the master configure definition file)
• mpiimpl.h (which includes mpidpre.h and mpidpost.h)
• mpidpkt.h and mpidi_ch3_mpid.h

Channel Layer

src/mpid/ch3/channels/sock/include/mpidi_ch3_impl.h is scoped to be used only within its specific channel (e.g., the sock channel). It includes channel-specific configurations like mpidi_ch3i_sock_conf.h and ch3usock.h.