Palace Documentation

Palace (Parallel Large-scale Computational Electromagnetics) is a 3D finite element solver for full-wave electromagnetic simulations. Key capabilities include:

• Simulation Domains: Frequency domain (with surface current excitation, lumped/numeric wave ports, and wideband sweeps) and Time domain (explicit or fully-implicit solvers).
• Eigenmode Calculations: Supports material/radiative loss and lumped impedance boundaries, with automatic postprocessing of energy-participation ratios (EPRs).
• Matrix Extraction: Extraction of lumped capacitance and inductance matrices via electrostatic and magnetostatic formulations.
• Mesh Management: Support for various mesh formats, built-in parallel mesh refinement, and Solution-based Adaptive Mesh Refinement (AMR) for frequency and time domain (except transient).
• High-Order Support: Arbitrary high-order finite element spaces and curvilinear meshes via the MFEM library.
• Hardware Acceleration: Support for NVIDIA and AMD GPUs (multi-GPU parallelism) using CUDA, HIP, and MAGMA.

Palace (PArallel LArge-scale Computational Electromagnetics) is an open-source, parallel finite element code designed for full-wave 3D electromagnetic simulations. It supports both frequency and time domain analysis and is built upon the MFEM finite element discretization library and libCEED for efficient exascale discretizations.

Core Capabilities

• Eigenmode Calculations: Includes optional material or radiative loss and lumped impedance boundaries. Supports automatic postprocessing of energy-participation ratios (EPRs) for circuit quantization and dielectric loss prediction.
• Frequency Domain Driven Simulations: Supports surface current excitation and lumped or numeric wave port boundaries. Features wideband frequency response calculation via uniform sampling or adaptive fast frequency sweeps.
• Time Domain Solver: Provides explicit or fully-implicit solvers for transient electromagnetic analysis.
• Electrostatic & Magnetostatic Formulations: Enables extraction of lumped capacitance and inductance matrices.
• 2D Waveguide Mode Analysis: Computes propagation constants, effective indices, and characteristic impedance on cross-sections (supports native 2D meshes or 3D submesh extraction).
• Mesh Management: Supports various mesh file formats, parallel mesh refinement (uniform or region-based), and solution-based Adaptive Mesh Refinement (AMR) for non-transient simulations.
• High-Order Accuracy: Utilizes arbitrary high-order finite element spaces and curvilinear mesh support via MFEM.
• Scalable Linear Solvers: Includes matrix-free $p$-multigrid (utilizing high-order operator partial assembly), parallel sparse direct solvers, and algebraic multigrid (AMG) preconditioners.
• Hardware Acceleration: Supports NVIDIA and AMD GPUs via CUDA, HIP, and MAGMA for multi-GPU parallelism.

Lumped ports allow you to model port boundaries using equivalent circuit impedances ($Z$). When specifying parameters, you must relate the desired circuit impedance ($Z$) to the surface element parameters ($R_s, L_s, C_s$) using a scaling factor $\alpha$.

1. Rectangular Ports

For rectangular ports, the surface impedance is related to the circuit impedance by: $Z = Z_s \frac{l}{w}$ where $l$ is the length (distance between conductors) and $w$ is the width.

Scaling factor $\alpha$: $\alpha = w/l$

Parameter mapping:

• $R_s = \alpha R$
• $L_s = \alpha L$
• $C_s = C/\alpha$

2. Coaxial Ports

For coaxial ports, the surface impedance is related to the circuit impedance by: $Z = Z_s \frac{\ln(b/a)}{2\pi}$ where $a$ is the inner radius and $b$ is the outer radius.

Scaling factor $\alpha$: $\alpha = \frac{2\pi}{\ln(b/a)}$

Parameter mapping:

• $R_s = \alpha R$
• $L_s = \alpha L$
• $C_s = C/\alpha$

Multielement Lumped Ports

If a port consists of multiple elements, their circuit impedances add in parallel: $\frac{1}{Z} = \sum_k \frac{1}{Z_k}$

For two identical elements, the effective impedance is $Z = Z_k / 2$.

During the CMake configuration step, Palace attempts to detect BLAS and LAPACK installations based on the system architecture:

x86_64 Systems

1. Intel MKL: Detected if MKLROOT environment variable is set.
2. AMD AOCL: Detected if AOCL_DIR or AOCLROOT environment variables are set (provides BLIS and libFLAME).
3. OpenBLAS: Attempted if the above are not found.

aarch64/arm64 Systems

1. Arm Performance Libraries (PL): Detected if ARMPL_DIR environment variable is set.
2. OpenBLAS: Attempted if the above is not found.

Customizing Detection

• To use a non-standard OpenBLAS installation, set OPENBLAS_DIR or OPENBLASROOT.
• Alternatively, add the installation path to CMAKE_PREFIX_PATH during the CMake call.

Note: It is recommended to use serial BLAS and LAPACK builds. Palace uses pure MPI parallelism for its standard parallelization approach.

To perform eigenmode simulations, set config["Problem"]["Type"] to "Eigenmode".

Requirements & Behavior:

• You must specify a nonzero frequency above which Palace will search for eigenmodes.
• Eigenvalues, mode quality factors, and errors (absolute and backward) are written to eig.csv in the directory specified by config["Problem"]["Output"].

Advanced Analysis:

• EPR Quantization: If you specify lumped ports corresponding to linearized lumped circuit elements, Palace automatically generates a participation matrix for inductive elements, saved as port-EPR.csv.
• Dissipative Elements: Specifying resistive lumped ports triggers coupling rate and quality factor calculations. The mode coupling quality factors ($Q_{ml} = \omega_m/\kappa_{ml}$) are written to port-Q.csv.

{
  "Problem": {
    "Type": "Eigenmode",
    "Output": "path/to/output",
    "Frequency": 1e9
  }
}

Palace uses three distinct unit systems during a simulation. Understanding these is critical for correctly interpreting mesh dimensions, solver behavior, and postprocessing results.

1. Mesh Units: The mesh has its own internal unit of length. You define the conversion from mesh units to meters using the config["Model"]["L0"] parameter.
2. Non-dimensional Units: The internal solver operates in a non-dimensional system where physical quantities are normalized by characteristic values (e.g., length, time, and power).
3. Postprocessing Units: Postprocessing uses a custom variant of SI units where the unit of time is nanoseconds (ns) instead of seconds. This is optimized for microwave engineering (GHz frequencies).

Important Considerations:

• Time Units: Because time is in nanoseconds, standard SI combinations like POWER * TIME do not equal ENERGY.
• Frequency: Internally, Palace uses angular frequency omega (though both omega and freq are used in the codebase).
• Field Scaling: When changing mesh units, fields defined on H(curl) and H(div) spaces require rescaling to maintain physical accuracy. See ScaleGridFunctions in PostOperator for implementation details.

{
  "Model": {
    "L0": 1e-6
  }
}

Palace provides a time-dependent formulation to compute the electric field response $\bm{E}(\bm{x},t)$ for a given time-dependent source excitation $\bm{U}^{inc}(\bm{x},t)$.

Governing Equations

The solver uses a second-order electric field differential equation: $$\nabla\times\mu_r^{-1}\nabla\times\bm{E} + \sigma\frac{\partial\bm{E}}{\partial t} + \varepsilon_r\frac{\partial^2\bm{E}}{\partial t^2} = 0$$

Implementation Details

• ODE System: The second-order equation is transformed into a first-order ODE system solved alongside the magnetic flux density $\bm{B}$.
• Integration: The formulation uses implicit adaptive time-stepping integration schemes.
• Efficiency: The $3 \times 3$ system can be block-eliminated to avoid expensive coupled block system solves while maintaining spatial discretization similarities with the frequency domain formulation.
• Boundary Conditions: For lumped resistive ports, the Robin boundary condition is expressed as: $$\bm{n}\times(\mu_r^{-1}\nabla\times\bm{E}) + Z_s^{-1}\bm{n}\times\left(\bm{n}\times\frac{\partial\bm{E}}{\partial t}\right) = \bm{U}^{inc}$$