NVSHMEM Documentation

NVSHMEM is an OpenSHMEM-based parallel programming interface designed for scalable NVIDIA GPU clusters. It provides a partitioned global address space (PGAS) across NVIDIA GPUs, enabling applications to perform one-sided transfers, atomics, signaling, synchronization, and collective operations.

NVSHMEM supports three primary interfaces:

• Host interface: For CPU-initiated communication.
• CUDA kernel interface: For GPU-initiated communication within kernels.
• CUDA stream interface: For stream-ordered communication.

The nvshmem crate provides Rust FFI bindings and high-level wrappers for the NVSHMEM host-side runtime. It manages process-global NVSHMEM state and symmetric memory allocation.

Core Components

• NvshmemRuntime: Initializes the process-global NVSHMEM host state using an explicit InitMethod.
• SymmetricBuffer<T>: Allocates and frees raw symmetric NVSHMEM memory. The buffer retains the NvshmemRuntime instance until the allocation is freed to ensure the runtime remains valid.
• sys / bindings: Provides the raw host FFI with exact nvshmem_* and nvshmemx_* names.
• Safe Wrappers: The crate re-exports prefix-stripped aliases and safe wrappers like my_pe, barrier_all, and int_put_on_stream.

Lifecycle Management

NVSHMEM finalizes only after the last NvshmemRuntime handle and all associated SymmetricBuffer instances have been dropped.

NVSHMEM follows semantic versioning using a MAJOR.MINOR.PATCH format. This determines API and ABI compatibility:

• MAJOR version update: Indicates backward-incompatible API or ABI changes.
• MINOR version update: Indicates backward-compatible API or ABI additions.
• PATCH version update: Indicates compatible bug fixes.

When performing vendor version checks, note that the CMake project version is always formatted as MAJOR.MINOR.PATCH.0 to ensure numeric field consistency for ABI and runtime checks.

The contrib/ directory contains community and partner contributions that extend NVSHMEM. These additions include language bindings, communication algorithms, higher-level APIs, tools, or reference implementations.

Key distinction:

• contrib/ content: Built on top of NVSHMEM's public APIs (nvshmem.h, nvshmemx.h, and nvshmem_host.h). It is maintained by its respective contributors, not the NVSHMEM core team, and does not follow NVSHMEM release quality standards.
• NVSHMEM core (src/): Managed by the core team. Changes to core files must follow the standard contribution process in CONTRIBUTING.md.

To initialize NVSHMEM using an MPI communicator, use InitMethod::MpiComm.

This requires a caller-provided pointer to an already initialized MPI_Comm. You must construct the method using unsafe { MpiComm::from_raw(...) }.

Note: The caller is responsible for ensuring that the MPI implementation matches the one used by NVSHMEM and that the communicator remains valid for the entire duration of the NVSHMEM runtime lifecycle.

If you only require raw host FFI declarations without CUDA-Oxide device bindings or the runtime crate, use the NVSHMEM_BUILD_RUST_HOST_ONLY=ON flag. This will generate only nvshmem_host.rs and nvshmem_host_api.rs.

Note: A CUDA Toolkit installation is still required because the NVSHMEM host headers include CUDA types.

cmake -S contrib/nvshmem4rust -B build/nvshmem4rust-host \
  -DNVSHMEM_SOURCE_DIR="$PWD" \
  -DNVSHMEM_HOME="$PWD/install" \
  -DNVSHMEM_CUDA_HOME=/path/to/cuda \
  -DNVSHMEM_BUILD_RUST_HOST_ONLY=ON

cmake --build build/nvshmem4rust-host --target build_bindings_rust

To successfully build and run applications using the nvshmem crate, configure the following environment variables:

Build Time

• NVSHMEM_HOST_LIB_DIR: Set this to the directory containing libnvshmem_host.so so Cargo can find the host library.
• NVSHMEM_CUDA_OXIDE_ROOT: Configured via CMake; the generated Cargo.toml uses this to point cuda-core to the correct location.

Runtime

• LD_LIBRARY_PATH: Must include the paths to NVSHMEM and its dependencies so the dynamic loader can resolve them.
• NVSHMEM_HOST_LIB_PATH: (Optional) Allows you to select the exact host-library path that the crate reopens using RTLD_GLOBAL.

After installation, the executables are organized under the directory specified by NVSHMEM_PERFTEST_INSTALL, following the source hierarchy. To run a specific test, use the corresponding binary path. For example, to run the point-to-point bandwidth test, use the shmem_put_bw binary located in the device/pt-to-pt/ subdirectory.

# Example: Running the point-to-point bandwidth test
# Replace $NVSHMEM_PERFTEST_INSTALL with your actual installation path
$NVSHMEM_PERFTEST_INSTALL/device/pt-to-pt/shmem_put_bw

To install NVSHMEM using a pre-downloaded package, follow the official installation guide. After installation, you must set the NVSHMEM_PREFIX environment variable to your installation directory and update LD_LIBRARY_PATH so the system can locate the libraries.

export NVSHMEM_PREFIX=/path/to/nvshmem
export LD_LIBRARY_PATH="$NVSHMEM_PREFIX/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

If you are using CUDA-Oxide, you must register each loaded module before launching kernels that utilize NVSHMEM device state.

Use NvshmemRuntime::register_module(&module) to obtain a registration guard. This guard ensures the runtime and the CUDA-Oxide module remain valid through NVSHMEM finalization.

Important: You should explicitly call .finalize() on the guard after your last synchronized kernel to report any finalization errors.

let registration = unsafe { runtime.register_module(&module) }?;

// launch CUDA-Oxide kernels that call NVSHMEM device functions

registration.finalize()?;