AMReX Documentation

AMReX is a C++ software framework designed for developing block-structured adaptive mesh refinement (AMR) algorithms. It is used to solve systems of partial differential equations (PDEs) with complex boundary conditions on current and emerging architectures (CPUs and GPUs).

Key capabilities include:

• Adaptive Mesh Refinement (AMR): Reduces computational cost and memory footprint by using a non-uniform mesh hierarchy.
• Multi-language Support: Core data structures and operations are in C++, but a Fortran-interface allows applications to be written entirely in Fortran.
• Interoperability: Provides interfaces to popular scientific libraries such as SUNDIALS, PETSc, and hypre.
• Complex Geometries: Supports the cut cell / embedded boundary approach for discretizing complex shapes using face apertures, normals, and volume fractions.
• Parallelism: Uses an MPI+X strategy. On CPUs, it typically uses MPI + OpenMP. On GPUs, it uses MPI + CUDA, while also supporting OpenACC and OpenMP for offloading subroutines.

AMReX is a software framework for massively parallel block-structured adaptive mesh refinement (AMR) applications, specifically designed for solving partial differential equations (PDEs) on hierarchical adaptive grids.

Key capabilities include:

• Multi-language support: C++ and Fortran interfaces.
• Data types: Support for cell-centered, face-centered, edge-centered, and nodal data.
• PDE Solves: Support for hyperbolic, parabolic, and elliptic solvers.
• Time-stepping: Optional subcycling in time for time-dependent PDEs.
• Physics features: Support for particles and embedded boundary descriptions for irregular geometry.
• Parallelization: Supports flat MPI, OpenMP, hybrid MPI/OpenMP, or MPI/MPI.
• GPU Acceleration: Backends available for CUDA (NVIDIA), HIP (AMD), or SYCL (Intel).
• I/O and Visualization: Parallel I/O with support for the Plotfile format, compatible with Amrvis, VisIt, ParaView, and yt.
• Profiling: Built-in profiling tools for performance analysis.

The amrex/Src/AmrCore directory provides the core components for Adaptive Mesh Refinement (AMR) in AMReX. It handles the fundamental data structures and operations required for multi-level grid simulations, including:

• Grid Management: Storing grid layout information and mapping processor distributions at each refinement level.
• Grid Creation: Functions to create grids at various levels, including support for tagging operations.
• Data Operations: Operators for interpolation (prolongation) and restriction between different refinement levels.
• Flux Management: Flux registers for storing and manipulating fluxes at coarse-fine interfaces.
• Particle Support: Integration for particles within an AMR framework.

Note: While amrex/Src/Amr/ contains additional classes for managing time-stepping, it is possible to implement fully adaptive, subcycling-in-time simulations using only the AmrCore components.

AMReX supports offloading work to the GPU via several backends: CUDA, HIP, SYCL, OpenACC, or OpenMP.

Portable C++ Launchers (CUDA, HIP, SYCL)

AMReX provides portable C++ function calls or macros that launch user-defined lambda functions.

• Without CUDA/HIP/SYCL: The lambda runs on the CPU.
• With CUDA/HIP/SYCL: The launch function automatically calculates blocks/threads, selects the appropriate stream or queue, and defines the work chunk for each thread.

Pragma-based Offloading (OpenACC, OpenMP)

Users can add appropriate pragmas to work loops and functions. These work with AMReX's internal CUDA-based memory management to ensure data is available on the GPU when the offloaded function executes.

Best Practices

AMReX recommends writing primary floating-point operation kernels using the Array4 object syntax. This provides a multi-dimensional array syntax (similar to Fortran) that maintains high performance.

AMReX provides performance portability across multiple GPU architectures using native programming languages: CUDA (NVIDIA), HIP (AMD), and SYCL (Intel).

Key Concepts

• MPI+X Model: AMReX uses an MPI+X approach. On GPUs, X is CUDA, HIP, or SYCL. Unlike CPU execution where tiling is used, GPU tiling is disabled by default to reduce kernel launch overhead. Instead, kernels are typically launched at the Box level, mapping one or more cells to each GPU thread.
• Work Distribution: GPU kernels are usually launched via ParallelFor looping constructs using GPU extended lambdas. These constructs launch kernels with a large number of threads, where each thread processes a small number of cells.
• Parallelism via Streams: AMReX utilizes GPU streams to maximize usage. Each iteration of an MFIter loop is placed on a separate stream, allowing independent iterations to run concurrently while maintaining execution order within a single stream.
• Memory Management: AMReX can use GPU managed memory to automatically handle data movement for mesh and particle data. While useful for initial porting, explicit host/device data migration is recommended for peak performance. Note that FArrayBox and MultiFab do not use managed memory by default.
• Performance Best Practice: Keep mesh and particle data on the GPU for as long as possible to minimize CPU-GPU transfer overhead. Only move data back to the CPU for I/O operations.
• Language Support: GPU support is available for C++. The Fortran interface does not currently support GPU offloading; porting Fortran code to C++ is recommended for GPU applications.