WarpX Documentation

WarpX is an advanced Particle-In-Cell (PIC) code designed for high-performance plasma physics simulations. It is highly parallel and optimized for multi-core CPUs as well as NVIDIA, AMD, or Intel GPUs, scaling to exascale supercomputers.

Key Features:

• Field Solvers: Supports Maxwell's equations, Poisson's equation, and Ampere's law coupled with Ohm's law.
• Geometries: Supports 1D, 2D, and 3D Cartesian, cylindrical, and spherical grids.
• Multi-physics: Includes ionization, atomic, fusion, collisional physics, and quantum electrodynamics (QED).
• Numerical Methods: Supports explicit and implicit time advancement, mesh refinement, boosted-frame simulations, embedded boundaries, and pseudo-spectral solvers.
• Extensibility: Can be extended via a Python interface to couple with other codes or AI/ML frameworks.
• Platforms: Runs on Linux, macOS, and Windows.

WarpX is an advanced electromagnetic and electrostatic Particle-In-Cell (PIC) code. It is designed for high performance and scalability, capable of running on GPUs and multi-core CPUs with built-in load balancing. It is suitable for large-scale simulations on supercomputers.

Key features include:

• Perfectly-Matched Layers (PML)
• Mesh refinement
• Boosted-frame technique
• Support for various high-performance computing (HPC) backends and data formats.

WarpX supports in situ visualization, which allows you to visualize simulation data directly from the running simulation without the overhead of dumping large data files to disk. This is achieved through integration with external visualization frameworks. Supported frameworks include:

• Ascent: For in situ visualization.
• Catalyst: For in situ visualization.
• Sensei: For in situ visualization.

Refer to the specific documentation for Ascent, Catalyst, or Sensei to learn how to configure and use these capabilities within your WarpX simulation.

WarpX generates several types of output data depending on the simulation requirements:

• Field and Particle Data:
  • openPMD: Supported via community formats like ADIOS and HDF5.
  • AMReX Plotfiles: The native AMReX data format.
• Reduced Diagnostics: On-the-fly observables (e.g., energy histograms, particle beam statistics) typically stored as tabular text (CSV) files.

WarpX source code is organized within the Source/ directory. Key components of the PIC (Particle-In-Cell) loop are located as follows:

• Main PIC loop: Implemented in WarpX::Evolve within Source/WarpXEvolve.cpp.
• Core PIC loop logic:
  • When subcycling is OFF: WarpX::OneStep_nosub.
  • When subcycling is ON (method 1): WarpX::OneStep_sub1.
• Main WarpX class: Implemented in Source/WarpX.cpp.

The following tutorial topics are currently available to help users understand different physics applications in WarpX:

• Magnetic mirror: Explores magnetic mirror physics.
• Beam-beam collision: Explores beam-beam collision scenarios.

Pinned (or "page locked") host memory is required for:

• Performing asynchronous copies between the host (CPU) and the device (GPU).
• Writing to CPU memory directly from a GPU kernel.

In WarpX/AMReX, pinned memory is typically provided via the amrex::PinnedArenaAllocator. This allocator is passed as the last argument to the constructors of ParticleContainer and MultiFab to enable optimized data transfer routines, such as output initialization.

HPC systems utilize batch (scheduling) systems to manage compute jobs asynchronously. WarpX documentation provides quick reference guides (cheat sheets) for interacting with the following common batch systems:

• Slurm: Used at NERSC, OLCF Frontier, etc.
• Flux: A modern resource manager framework used at LLNL LC, etc.
• LSF (Load Sharing Facility): An IBM batch system used at OLCF Summit, LLNL Lassen, etc.
• PBS (Portable Batch System): Includes OpenPBS, PBS Pro, and TORQUE.
• PJM (Parallel Job Manager): A Fujitsu batch system used at RIKEN Fugaku, etc.

Individual HPC machine guides typically include job submission example scripts that can be used as templates for your own compute jobs.

SENSEI is a framework for in situ data analysis that provides uniform access to various visualization and analysis back ends (e.g., VisIt Libsim, ParaView Catalyst, VTK-m, Ascent, ADIOS, Yt, and Python) via a runtime XML configuration.

Its architecture consists of three main components:

1. Data Adaptors: Present simulation data in the SENSEI data model.
2. Analysis Adaptors: Present back-end data consumers to the simulation.
3. Bridge Code: Manages adaptors and periodically pushes data through the system.

Users can switch between different back ends (like Catalyst or Libsim) by simply changing the XML configuration file without modifying the simulation source code.