DOCSEARCHbyAlgolia
Submit Docs

ADIOS2: The Adaptable Input Output System version 2

repository·master·Indexed 43 days ago

https://github.com/ornladios/adios2

A profiling interface for libraries and applications providing high-performance IO capabilities. ADIOS2 supports multiple engines including BP, DataMan, HDF5, SST, Inline, and Skeleton, with APIs for C++, C, Fortran, Python, and Julia. It features support for global, joined, and local arrays, as well as GPU acceleration via CUDA, HIP, and Kokkos.

Tokens
165.9K
Snippets
484
Records
697
Agent score
53%

What's inside ADIOS2

  1. Overview of supported ADIOS 2 command line tools

    master

    The following command line utilities are currently supported by ADIOS 2:

    • bpls: Used for exploring bp and hdf5 file data and metadata in human-readable formats.
    • adios_reorganize: Tool for data reorganization.
    • adios2-config: Configuration utility.
    • sst_conn_tool: A diagnostic tool for checking SST (SST staging engine) connectivity.

  2. Overview of the Googletest Mocking (gMock) Framework

    master

    gMock is a C++ framework for creating and using mock classes. It is designed to help improve system design and testing quality by providing a declarative syntax for defining mocks and controlling their behavior.

    Key features include:

    • Declarative Mock Definition: Define mock objects using a specialized syntax.
    • Partial (Hybrid) Mocks: Create objects that combine real implementation with mocked behavior.
    • Argument Validation: Use a rich set of matchers to validate function arguments.
    • Behavior Control: Intuitive syntax for specifying what a mock should do when called.
    • Automatic Verification: Expectations are verified automatically (no manual record-and-replay required).
    • Ordering Constraints: Express arbitrary (partial) ordering constraints on function calls.
    • Extensibility: Users can define custom matchers and actions.
    • C++ Specifics: Handles overloaded functions and arbitrary types without using exceptions.

  3. Overview of ADIOS2 Adaptable IO

    master

    ADIOS2 is an adaptable, scalable, and unified framework designed for scientific data management at scale. It is intended for use cases where data transfer volumes exceed the capabilities of traditional file I/O.

    Key capabilities include:

    • Custom Data Management: Handles massive datasets across their lifecycle (generation, analysis, movement, and storage).
    • Rapid Metadata Extraction: Uses self-describing data in binary-packed (.bp) format.
    • Data Extraction: Ability to separate and extract relevant information from large datasets.
    • Real-time Analytics: Supports real-time decision-making based on in-transit or in-situ analytics.
    • Extensible Transport: Can expand to wide area networks, remote direct memory access (RDMA), and shared memory with minimal overhead.
    • Hardware Optimization: Utilizes emergent hardware technologies like high-bandwidth memory and burst buffers.

  4. Overview of adios_reorganize and adios_reorganize_mpi

    master

    adios_reorganize (serial) and adios_reorganize_mpi (MPI) are generic ADIOS tools used to read data from an existing ADIOS stream and output it into a new ADIOS stream.

    Common use cases include:

    • Format Conversion: Converting files between ADIOS BP and HDF5 formats (requires ADIOS to be built with HDF5 support).
    • I/O Staging: Using separate compute nodes to stage I/O to/from disk to prevent application bottlenecks.
    • Data Reorganization: Changing the number of data blocks in a file (e.g., reducing the number of blocks from a large-scale application to make subsequent reads faster).

  5. Overview of nanobind

    master

    nanobind is a lightweight C++ library designed to expose C++ types to Python and vice versa. It is intended as a high-performance successor to pybind11 and Boost.Python, utilizing a near-identical syntax while providing significant improvements in efficiency.

    Key advantages include:

    • Faster Compilation: Up to ~4× faster compile times compared to pybind11.
    • Smaller Binaries: Up to ~5× smaller binaries than pybind11 and 3-12× reduction compared to Cython.
    • Better Runtime Performance: Up to ~10× lower runtime overheads compared to pybind11.
    • Stable ABI Support: Ability to target the Python Stable ABI (starting with Python 3.12), which reduces the need to ship per-Python version plugins.

  6. What is ADIOS2?

    master

    ADIOS2 (Adaptable Input Output System) is a unified, high-performance I/O framework designed for extreme-scale scientific data. It provides a consistent abstraction API to transport and transform groups of self-describing data variables and attributes across various media, including files, wide-area networks, in-memory staging, and more.

    Key characteristics include:

    • Streaming-oriented: The framework favors asynchronous transfers of variable groups. Synchronous, single-variable transfers are treated as special cases.
    • Step-based: Data is organized into "steps" representing groups of variables, suitable for both streaming and random-access (file) media.
    • MPI-compatible: Supports both parallel MPI applications and serial codes.
    • Scalable: Designed for supercomputer applications handling petabytes of data.
    • Multi-language support: While native in C++, it provides bindings for Python, C, Fortran, and Matlab.

  7. Overview of the perfstubs Profiling Interface

    master

    perfstubs is a thin stub interface designed for instrumenting library or application code for performance profiling. It is designed to be extremely low-overhead: if no measurement library (like Score-P or TAU) is loaded, the function calls are essentially no-ops (a few instructions) because they operate on function pointers initialized to nullptr.

    Key Features:

    • Zero-cost when unused: If function pointers are nullptr, the overhead is minimal.
    • Flexible Linking: Supports both dynamic linking (using dlsym() to find symbols at runtime via LD_PRELOAD) and static linking (using a weak symbol technique).
    • Convenience Macros: Provides macros to automatically generate descriptive timer names using source code metadata like __FILE__, __LINE__, and __func__.
    • C++ Scoped Timers: Includes RAII-style scoped timers to ensure timers are stopped automatically when a function returns or an exception is thrown.

  8. Overview of the ADIOS2 XRootD Server Plugin

    master

    The ADIOS2 XRootD Server Plugin is a server-side component designed to serve remote reads of ADIOS2 (BP5) datasets over XRootD SSI and HTTP(S) interfaces. It acts as the data-plane backend for remote campaign access.

    It consists of two primary libraries:

    1. adios2_xrootd: The SSI service responsible for request processing and managing the file pool.
    2. adios2_xrootd_http: An HTTP-to-SSI bridge handler.

    Note: The _adios segment is a reserved name in the URL path. No served dataset or file should use _adios as a path component to avoid routing conflicts.

  9. Overview of ADIOS2

    master

    ADIOS2 (The Adaptable Input Output System version 2) is a scalable and unified framework for scientific data I/O. It provides a single API to transport self-describing variables and attributes across files, networks, and memory.

    Key capabilities include:

    • Unified I/O API: Switch between files, wide-area networks, and in-memory staging without changing application code.
    • BP format: A self-describing binary-packed format (.bp) with rapid metadata extraction and compression.
    • Scalability: MPI-native support for petabyte-scale workloads, while also supporting serial use.
    • Streaming/Steps: Asynchronous, grouped variable transport using an explicit 'steps' abstraction.
    • Extensibility: Plugin support for custom engines and operators via shared-library plugins at runtime.
    • HDF5 VOL: Allows existing HDF5 applications to use ADIOS2 as a transparent backend.

  10. Overview of PerfStubs

    master
    PerfStubs is a profiling API designed to add support for external tool instrumentation to any project. It provides a way to instrument code without introducing direct build dependencies on specific measurement tools. While originally prototyped for TAU, it is now a generic interface used for testing and demonstration purposes.

  11. Understand the Lorenz System Example capabilities

    master

    The Lorenz system example demonstrates how ADIOS2 handles specific high-performance I/O challenges:

    • Independent Data Writing: Demonstrates how to write data that is completely independent of other data rapidly, which is useful for scaling in parallel environments.
    • Complex Data Structures: Shows how to store and manage non-matrix data structures. Specifically, it uses a std::vector<std::array<Real, 7>> to represent the phase-space state and derivatives, which is a non-trivial layout for I/O.
    • Parallelism without Coordination: The example is designed such that each MPI rank solves the Lorenz system with a different initial condition without requiring coordination between ranks, showcasing ADIOS2's ability to handle decoupled parallel workloads.

  12. Learn ADIOS2 IO capabilities via hello examples

    master

    The following list outlines the available hello examples and the specific capabilities they demonstrate. Use these to learn how to interact with different ADIOS2 engines and programming languages.

    Core IO and BP Engine Examples

    • helloWorld: Writing and reading a simple word using the BP engine. (C++, C, Python)
    • bpWriter: Writing a variable using the BP engine. (C++, C, Fortran, Python)
    • bpReader: Reading 1D, 2D, or 3D variables using the BP engine. (C++, Fortran, Python)
    • bpAttributeWriterReader: Writing and reading attributes using the BP engine. (C++)
    • bpOperatorSZWriter: Writing variables using the SZ operator with the BP engine. (C++)
    • bpStepsWriteRead: Writing and reading multiple steps using the BP engine. (C++)
    • bpFlushWriter: Flushing a variable using the BP engine. (C++)
    • bpFWriteCRead: Cross-language 2D variable handling (Fortran write/C++ read and vice versa) using the BP engine. (C++, Fortran)

    Accelerator and Parallelism Examples

    • bpStepsWriteReadCuda: Multi-step variable IO leveraging CUDA. (C++)
    • bpStepsWriteReadHip: Multi-step variable IO leveraging HIP. (C++)
    • bpStepsWriteReadKokkos: Multi-step variable IO leveraging Kokkos. (C++)

    Specialized Engines

    • DataMan Engine: Real-time WAN streams.
      • datamanReader: Reading variables. (C++, Python)
      • datamanWriter: Writing variables. (C++, Python)
    • DATASPACES Engine:
      • dataspacesReader: Reading variables. (C++)
      • dataspacesWriter: Writing variables. (C++)
    • HDF5 Engine:
      • hdf5Reader: Reading variables. (C++)
      • hdf5Writer: Writing variables. (C++)
      • hdf5SubFile: Parallel HDF5 using the subfile feature. (C++)
    • Inline Engine:
      • inlineMWE: Simple write/read. (C++)
      • inlineFWriteCppRead: Cross-language 2D variable handling (Fortran/C++). (C++, Fortran)
      • inlineReaderWriter: Writing two variables (including timestep) using time aggregation. (C++)
    • SST Engine:
      • sstReader: Reading variables. (C++)
      • sstWriter: Writing variables. (C++)
    • Skeleton Engine:
      • skeleton: Writing and reading a variable using a skeleton engine. (C++)