Parthenon AMR Framework

Parthenon is a performance-portable, block-structured Adaptive Mesh Refinement (AMR) framework. It is designed to minimize host-device overhead and maximize performance on modern hardware through several key architectural choices:

• Device-Resident Approach: Data is kept in device memory whenever possible to prevent expensive host-to-device transfers.
• Data Packing: Transparently packs data across blocks to reduce and hide kernel launch latency.
• Asynchronous Communication: Supports direct device-to-device communication via asynchronous MPI.
• Abstraction Layer: Provides an intermediate layer to simplify the complexity of device kernel launches.
• Extensibility: Features a flexible, plug-in package system.
• Flexible Data Models: Supports particles as well as cell-, node-, face-, and edge-centered fields.
• Metadata Control: Abstract variables are controlled via metadata flags.
• Parallelism: Uses multi-stage drivers/integrators with support for task-based parallelism.

In Parthenon, a package is implemented as a C++ namespace. To create a functional package, you must include an Initialize function, which returns a std::shared_ptr<StateDescriptor>. You will also typically include one or more tasks (functions that perform work, such as updating fields or particles).

Key components of a package include:

• Initialize function: Configures the package, parses input parameters, and registers fields or swarms.
• StateDescriptor: An object that tells Parthenon what variables (fields or params) the package requires.
• PAR_VAR macro: Creates a type-safe variable name (e.g., Ellipse.Indicator) that can be accessed on GPUs via string-like names without the performance penalty of actual strings. This also allows compile-time typo checking.

namespace MyPackage {
  using namespace parthenon::package::prelude;

  // Type-safe variable name
  PAR_VAR(MyPackage, MyVariable);

  std::shared_ptr<StateDescriptor> Initialize(ParameterInput *pin);
  TaskStatus MyTask(MeshData<Real> *md, const Real dt);
}

While Kokkos generally discourages using a View of Views, Parthenon sometimes requires this pattern. To ensure that inner View deallocators are called on the host (rather than the device), you must use parthenon::ViewOfViewAlloc to add the Kokkos::SequentialHostInit property.

When creating a ParArray1DRaw to represent a View of Views, use the following pattern:

ParArray1DRaw<ParArray1D<Real>> view_of_pararrays(parthenon::ViewOfViewAlloc("myname"), 10);

To create a host mirror of a View of Views, do not use Kokkos::create_mirror_view manually with SequentialHostInit. Instead, use the Parthenon-provided interface:

auto view_of_pararrays_h = create_view_of_view_mirror(view_of_pararrays);

// Correct way to allocate a View of Views
ParArray1DRaw<ParArray1D<Real>> view_of_pararrays(parthenon::ViewOfViewAlloc("myname"), 10);

// Correct way to mirror a View of Views to host
auto view_of_pararrays_h = create_view_of_view_mirror(view_of_pararrays);

By default, Parthenon assumes face positions in X1, X2, and X3 correspond to the x, y, and z components of mesh nodes. For moving meshes or coordinate transformations, you must explicitly provide coordinates using the Metadata::CoordinatesVec flag.

To implement this:

1. Declare a node-centered 3-vector variable in your Initialize function using the Metadata::CoordinatesVec flag.
2. Use a function (e.g., assigned to InitMeshBlockUserData) to populate this variable with the actual x, y, and z nodal positions.

Note: Non-standard coordinates are not supported in XDMF for 1D meshes; Parthenon will revert to traditional output in those cases.

// 1. Declare the field with Metadata::CoordinatesVec
pkg->AddField("locations",
  Metadata({
    Metadata::Node, 
    Metadata::CoordinatesVec, 
    Metadata::Derived, 
    Metadata::OneCopy
  }, std::vector<int>{3}));

// 2. Populate the field in a geometry setup function
void SetGeometryBlock(MeshBlock *pmb, ParameterInput *pin) {
  parthenon::par_for(DEFAULT_LOOP_PATTERN, "positions", DevExecSpace(), 0, pack.GetNBlocks() - 1, 
    KOKKOS_LAMBDA(const int b, const int k, const int j, const int i) {
      const auto &coords = pack.GetCoordinates(b);
      pack(b, 0, k, j, i) = coords.X<X1DIR, parthenon::TopologicalElement::NN>(k, j, i);
      pack(b, 1, k, j, i) = coords.X<X2DIR, parthenon::TopologicalElement::NN>(k, j, i);
      pack(b, 2, k, j, i) = coords.X<X3DIR, parthenon::TopologicalElement::NN>(k, j, i);
    });
}

Scratch indexing behavior depends on the loop tag used, as different tags visit different sets of cells. Scratch is always allocated as a contiguous memory-flat span [span_start, span_stop].

Loop Tag Behaviors

• bvoi: Enumerates contiguous flat spans and converts each back to (k, j, i). It sweeps the entire rectangular halo-extended box (including multi-axis corner cells). Scratch is sized over the full box from the low corner to the high corner.
• bovi: Visits only the union of shifted copies (contiguous flat spans). Scratch is sized over the tighter enclosing memory-flat interval of that union.
• boiv: A point-wise backend where the base range S is a single logical point. Scratch may specialize to compact per-cell local storage (e.g., Real scratch[2]) rather than a memory-flat span.

Indexing Logic

• Flat-index bodies: The index passed to the body is already relative to the current inner range's memory origin. Access is: idx - scratch_index_start.
• Coordinate bodies: The index (k, j, i) is first mapped to the memory-flat indexer, then the span start is subtracted: memory_kji.GetFlatIdx(k, j, i) - scratch_flat_start.

Parthenon provides several wrappers to simplify Kokkos usage:

• par_for wrappers: These use inclusive bounds, meaning the loop includes the last index provided.
• ParArrayND: Arrays that allocate on the device by default using the configured precision and include a State object for metadata.
• ParArray#DRaw: These map directly to Kokkos Views allocated on the device.
• DeviceAllocate and DeviceCopy: These return a std::unique_ptr to an object in device memory. The DeviceCopy variant also copies data from host memory. These automatically free device memory when they go out of scope, but they are not performant and should be used minimally.

Creating Host Mirrors

To create an array on the host with the same layout as a device array, use one of the following:

• auto arr_host = Kokkos::create_mirror(arr_dev);: Always creates a new array, even if the device and host share the same space (e.g., OpenMP).
• auto arr_host = Kokkos::create_mirror_view(arr_dev);: Creates a host array if HostSpace != DeviceSpace, or returns a reference to the existing array if HostSpace == DeviceSpace.

// Always create a new array
auto arr_host = Kokkos::create_mirror(arr_dev);

// Create host array or get reference if spaces are identical
auto arr_host = Kokkos::create_mirror_view(arr_dev);

You can create a custom unit system by defining a struct of conversion factors from SI units. These factors should ideally be constexpr. The PhysicalConstants class uses these factors as a template parameter to scale its internal SI-based hardcoded values to your desired system.

Required conversion factors in your struct:

• length (meter)
• mass (kilogram)
• time (second)
• temperature (Kelvin)
• current (Ampere)
• charge (Coulomb)
• capacitance (Farad)
• angle (Radian)

struct SIButKeVTemperatures {
  static constexpr double length = 1.;      // meter
  static constexpr double mass = 1.;        // kilogram
  static constexpr double time = 1.;        // second
  static constexpr double temperature = 8.6173e-8; // keV
  static constexpr double current = 1.;     // Amp
  static constexpr double charge = 1.;      // Coulomb
  static constexpr double capacitance = 1.; // Farad
  static constexpr double angle = 1.;       // Radian
};

// Usage with the custom system
parthenon::constants::PhysicalConstants<SIButKeVTemperatures> pc;
std::cout << "Boltzmann constant: " << pc.kb << std::endl;

The internal hierarchy of the coalesced communication system is as follows:

• Mesh::pcoalesced_comms: A std::shared_ptr to a CoalescedComms object stored on the Mesh.
• CoalescedComms: Contains maps from MPI rank and BoundaryType (e.g., regular communication, flux correction) to CoalescedBuffersRank objects.
• CoalescedBuffersRank: Manages send/receive rank pairs and contains maps from the sending MeshData partition ID (used as the MPI tag) to CoalescedBuffer objects.
• CoalescedBuffer: Handles the actual packing, sending, receiving, and unpacking. It owns the CommunicationBuffer<BufArray1D<Real>> used for the coalesced data and the allocation status message.

Sparse Awareness: Coalesced buffers are sparse-aware and dynamically allocate space based on allocated fields. To balance memory consumption and allocation frequency, Parthenon currently allocates twice the required memory whenever the total size of allocated variable-boundary buffers exceeds the current coalesced buffer size.