DOCSEARCHbyAlgolia
Submit Docs

SuperLU_DIST

repository·master·Indexed 42 days ago

https://github.com/xiaoyeli/superlu_dist

A high-performance parallel solver for large-scale sparse linear systems (A*X=B). Optimized for distributed memory architectures using MPI and OpenMP, it features Gaussian elimination with static pivoting (GESP) and supports GPU acceleration for Nvidia, AMD, and Intel GPUs. It provides support for double-precision real and complex data types, as well as a Fortran 90 wrapper interface.

Tokens
4.7K
Snippets
13
Records
20
Agent score
52%

What's inside SuperLU_DIST

  1. What is SuperLU_DIST?

    master

    SuperLU_DIST is a parallel extension of the serial SuperLU library designed for distributed memory parallel machines. It provides subroutines to solve sparse linear systems of the form A*X=B using Gaussian elimination with static pivoting (GESP).

    Key technical characteristics:

    • Static Pivoting (GESP): Combines the numerical stability of partial pivoting with the scalability of Cholesky (no pivoting) to ensure accuracy and efficiency on large processor counts.
    • Parallelism Model: Implemented in ANSI C, utilizing OpenMP for on-node (shared memory) parallelism and MPI for off-node (distributed memory) communications.
    • GPU Acceleration: Supports numerical LU factorization and triangular solvers on Nvidia, AMD, and Intel GPUs.
    • Data Types: Supports double-precision real and double-precision complex data types.

  2. Configure SuperLU_DIST performance and execution via environment variables

    master

    Several environment variables control parallel execution, GPU offloading, and solver performance parameters.

    Execution and Parallelism

    • OMP_NUM_THREADS=<number>: Sets the number of OpenMP threads.
    • SUPERLU_ACC_OFFLOAD=1: Enables GPU offloading (default is 1).

    Solver Performance Parameters

    These variables affect integer blocking parameters and can be queried via SRC/sp_ienv.c:

    • NREL=<value>: Supernode relaxation parameter.
    • NSUP=<value>: Maximum allowable supernode size (not to exceed 512).
    • FILL=<value>: Estimated fill ratio of nonzeros (L+U)/nonzeros(A).
    • MAX_BUFFER_SIZE=<value>: Maximum buffer size on GPU for GEMM.
    export OMP_NUM_THREADS=4
export NREL=10
export NSUP=256

  3. Understand the SuperLU_DIST directory structure

    master

    The source code is organized as follows:

    • SRC/: C source code compiled into libsuperlu_dist.a.
    • lib/: Contains the library archive libsuperlu_dist.a.
    • EXAMPLE/: Example programs (see EXAMPLE/README for running instructions).
    • DOC/: The Users' Guide.
    • FORTRAN/: Fortran90 wrapper functions.
    • TEST/: Testing code.
    • CBLAS/: BLAS routines in C (Note: this version is single-threaded; performance with OpenMP depends on your multithreaded BLAS implementation).
    • INSTALL/: Test machine-dependent parameters.
    • MAKE_INC/: Sample machine-specific make.inc files.
    • Makefile: Top-level Makefile for installation and testing.
    • make.inc: Compiler, flags, and library definitions used by all Makefiles (may require manual editing).

  4. Install SuperLU_DIST using CMake

    master

    To install SuperLU_DIST using the CMake build system, create a build tree and configure it with the necessary third-party library paths.

    Prerequisites

    • ParMETIS: Required for parallel symbolic factorization. You must set PARMETIS_ROOT and PARMETIS_BUILD_DIR environment variables.
    • BLAS: You can link an existing BLAS library or use the internal (unoptimized) CBLAS.
    • Optional Libraries:
      • CombBLAS: For parallel weighted matching (HWPM) pre-pivoting.
      • LAPACK: To improve speed in triangular solve routines.
      • GPU (CUDA/HIP): For NVIDIA or AMD GPU acceleration.

    Installation Steps

    1. Create a build directory and enter it.
    2. Run cmake with the required definitions (see examples below).
    3. Run make to compile.
    4. Run make install to install the libraries.
    5. Run ctest to verify the installation.
    # Simple installation (requires ParMETIS)
mkdir build && cd build
cmake .. \
    -DTPL_PARMETIS_INCLUDE_DIRS="${PARMETIS_ROOT}/include;${PARMETIS_ROOT}/metis/include" \
    -DTPL_PARMETIS_LIBRARIES="${PARMETIS_BUILD_DIR}/libparmetis/libparmetis.a;${PARMETIS_BUILD_DIR}/libmetis/libmetis.a"

make
make install
ctest

  5. Install SuperLU_DIST using Makefile

    master

    Manual installation is performed by editing the make.inc file. This file centralizes machine-specific information used by all subdirectories.

    Setup Steps

    1. Select a template: Copy a sample make.inc from the MAKE_INC/ directory to your root directory (e.g., cp MAKE_INC/make.xt5 make.inc).
    2. Configure BLAS:
      • For vendor BLAS: Set BLASDEF = -DUSE_VENDOR_BLAS and BLASLIB = <library_name>.
      • For internal CBLAS: Comment out BLASDEF and set BLASLIB = ../lib/libblas$(PLAT).a, then run make blaslib.
    3. Configure External Libraries (in make.inc):
      • ParMETIS: Set HAVE_PARMETIS = TRUE, METISLIB, PARMETISLIB, and I_PARMETIS paths.
      • LAPACK: Set SLU_HAVE_LAPACK = TRUE and LAPACKLIB.
      • CombBLAS: Set HAVE_COMBBLAS = TRUE, COMBBLASLIB, and I_COMBBLAS paths.
    4. Build: Run make at the top level.
    # Example: Setting up ParMETIS in make.inc
HAVE_PARMETIS = TRUE
METISLIB = -L<metis directory> -lmetis
PARMETISLIB = -L<parmetis directory> -lparmetis
I_PARMETIS = -I<parmetis directory>/include -I<parmetis directory>/metis/include

  6. Install SuperLU_DIST via CMake

    master

    The recommended installation method is using the CMake automatic build system. This method handles dependencies and build configurations automatically.

    Dependencies:

    • Required: BLAS and ParMETIS.
    • Optional: CombBLAS and LAPACK.
    • GPU Support: Can be enabled during the CMake configuration process.

  7. Install SuperLU_DIST on Windows

    master

    To install SuperLU_DIST on Windows, you need CMake, Visual Studio, and Microsoft HPC Pack. The following steps use a CMake configuration that has been tested with Visual Studio 2017, without ParMETIS, without Fortran, and with OpenMP disabled.

    1. Configure the build using CMake with the appropriate flags for your environment.
    2. Build and install using jom or nmake.
    3. Libraries will be installed to the path specified by CMAKE_INSTALL_PREFIX under the /lib subdirectory.
    # 1. Configure
'/winsame/contrib-vs2017/cmake-3.9.4-ser/bin/cmake' \
  -DCMAKE_INSTALL_PREFIX:PATH=C:/winsame/volatile-vs2017/superlu_dist-master.r147-parcomm \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DCMAKE_COLOR_MAKEFILE:BOOL=FALSE \
  -DCMAKE_VERBOSE_MAKEFILE:BOOL=TRUE \
  -Denable_openmp:BOOL=FALSE \
  -DCMAKE_C_COMPILER:FILEPATH='C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.11.25503/bin/HostX64/x64/cl.exe' \
  -DCMAKE_C_FLAGS:STRING='/DWIN32 /D_WINDOWS /W3' \
  -DTPL_ENABLE_PARMETISLIB:BOOL=FALSE \
  -DXSDK_ENABLE_Fortran=OFF \
  -G 'NMake Makefiles JOM' \
  C:/path/to/superlu_dist

# 2. Build and Install
jom # or nmake
jom install  # or nmake install

# 3. Test (optional)
ctest

  8. Use the Fortran 90 Interface for SuperLU_DIST

    master

    SuperLU_DIST provides a Fortran 90 wrapper interface to access its data structures and routines. The interface is composed of several key files:

    • superlu_mod.f90: The Fortran 90 module defining wrapper functions for SuperLU_DIST data structures.
    • superlupara.f90: Contains parameters corresponding to SuperLU_DIST's enumerated constants.
    • superlu_c2f_wrap.c: C wrapper functions that allow C-based SuperLU_DIST routines to be called from Fortran.
    • hbcode1.f90: A Fortran routine specifically for reading matrices in the Harwell-Boeing format.

    To compile the Fortran interface, use the make command within the directory.

  9. Understand SuperLU_DIST solver reuse patterns

    master

    SuperLU_DIST provides different solver variants to optimize performance by reusing previously computed information:

    VariantFunctionality
    pddrive / pzdriveStandard solve using full default options.
    pddrive1 / pzdrive1Reuse Factored Form: Solves systems with the same matrix $A$ but different right-hand sides.
    pddrive2 / pzdrive2Reuse Sparsity Ordering: Solves systems with the same sparsity pattern of $A$.
    pddrive3 / pzdrive3Reuse Sparsity Ordering and Row Pivoting: Solves systems with the same sparsity pattern and similar values.
    pddrive3d3D Algorithms: Uses communication-avoiding 3D algorithms, supports GPU offload, and provides a batch interface (e.g., PxGSSVX3D or PxGSSVX_CSC_BATCH).

  10. Install the SuperLU_DIST Python interface

    master

    To compile and install the Python interface, you must use the CMake build system with specific flags and ensure mpi4py is installed.

    1. Install dependencies: Install mpi4py version 4.0.0 or higher using the MPI compiler:

      MPICC=mpicc pip install mpi4py==4.0.0

    2. Configure and Build: Use CMake with -Denable_python=ON (enabled by default) and -DBUILD_SHARED_LIBS=ON. After configuring, run the following command to build the shared library superlu_dist_python:

      make python
    # Install mpi4py
MPICC=mpicc pip install mpi4py==4.0.0

# Build the Python interface
cmake -Denable_python=ON -DBUILD_SHARED_LIBS=ON ..
make python

  11. Use bash scripts for automated testing

    master

    For running multiple test iterations with varying input parameters, use the provided bash scripts:

    • pdtest.sh: Automates multiple runs for real-valued tests.
    • pztest.sh: Automates multiple runs for complex-valued tests.
    ./pdtest.sh
./pztest.sh

  12. Run SuperLU_DIST tests

    master

    You can run testing programs to verify SuperLU_DIST functions using pdtest for real-valued matrices or pztest for complex-valued matrices. These tests require an MPI environment and specific flags to define the process grid and input files.

    Required flags:

    • -r <process row>: Specifies the number of rows in the process grid.
    • -c <process columns>: Specifies the number of columns in the process grid.
    • -f <file>: Path to the input file (e.g., .rua for real or .cua for complex).

    Note: The total number of MPI processes (-n <np>) must equal the product of the rows and columns (<process row> * <process columns>).

    # Run real-valued tests
mpiexec -n <np> pdtest -r <process row> -c <process columns> -f ../EXAMPLE/g20.rua 

# Run complex-valued tests
mpiexec -n <np> pztest -r <process row> -c <process columns> -f ../EXAMPLE/cg20.cua