Open MPI Documentation

The MEMHEAP infrastructure is responsible for managing the symmetric heap in OpenSHMEM. It provides a framework for memory allocation on the symmetric heap through different components.

Currently supported components include:

• buddy: Uses a buddy allocator to manage memory allocations on the symmetric heap.
• ptmalloc: An adaptation of ptmalloc3.

The infrastructure automatically selects the component with the highest priority during initialization.

OpenSHMEM provides a low-latency, high-bandwidth communication API for highly parallelized scalable programs. It follows a Single Program, Multiple Data (SPMD) style where multiple processes, called Processing Elements (PEs), run the same program simultaneously.

Key capabilities include:

• Remote Data Transfer: Using put operations (to send data to a remote PE), get operations (to retrieve data from a remote PE), and remote pointers.
• Collective Operations: Broadcast, reduction, and barrier synchronization.
• Atomic Memory Operations: Atomic read-and-update operations (e.g., fetch-and-increment) on local or remote data objects.

The SHMEM API can be used independently or in combination with MPI routines within the same parallel program.

The BTL/OFI component is an RDMA-only Byte Transfer Layer (BTL) based on the Libfabric (OFI) API. Its primary purpose is to enable RDMA across multiple vendor hardware through a single interface. Most high-level operations are managed by the upper layers (osc/rdma), while this BTL handles low-level communication tasks.

Tested providers include:

• sockets (Proof of Concept, slow, potentially unstable)
• psm2 (High performance)

Open MPI's support for fault tolerance is categorized as follows:

Currently Supported

• User Level Fault Mitigation (ULFM): The primary supported technique for resilience.

Research / Non-Production Only

• Message logging techniques: Similar to MPICH-V implementations.

Deprecated / No Longer Available

• Coordinated and uncoordinated process checkpoint and restart: (Formerly similar to LAM/MPI and MPICH-V).
• Data Reliability and network fault tolerance: (Formerly similar to LA-MPI).

Parallel debugging in Open MPI can be approached using two main categories of tools:

1. Debuggers:

   • Serial Debuggers: Standard debuggers like gdb that operate on individual processes.
   • Parallel Debuggers: Tools that attach to all individual processes in an MPI job simultaneously. They treat the entire MPI application as a single entity, allowing for control over the whole job, manual replication of race conditions, and other high-level abstractions.

2. Profilers:

   • Tools that analyze MPI usage and provide statistics and meta-information about the application run.
   • Live Profilers: Present information as the application runs.
   • Post-mortem Profilers: Collect information during the run for analysis after the application has finished.

Open MPI is organized into distinct projects that act as abstraction barriers. These projects are compiled into separate libraries with a specific dependency order. When linking MPI executables, you typically link against libmpi, which implicitly pulls in its dependencies like libopen-pal.

Dependency Order:

1. OSHMEM depends on OMPI
2. OMPI depends on OPAL

Note that while these are organized by dependency, they are not strict functional layers; for performance reasons, higher-level code (like OMPI) may call the operating system or network interfaces directly without traversing the lower-level libraries.

MPI processes in Open MPI can be launched in two main types of environments:

1. Scheduled environments: Systems that use a resource manager or scheduler to control access to compute nodes (e.g., Slurm, PBS/Pro/Torque, and LSF).
2. Non-scheduled environments: Systems without a resource manager. These are typically local launches (e.g., on a single workstation) or launches across a small number of nodes via ssh.