parasail

repository·master·Indexed 43 days ago

https://github.com/jeffdaily/parasail

A high-performance SIMD C library for pairwise sequence alignment implementing Smith-Waterman, Needleman-Wunsch, and semi-global algorithms. It features runtime CPU dispatching for SSE2, SSE4.1, AVX2, AltiVec, and NEON, and includes the parasail_aligner CLI tool for comparing FASTA/FASTQ files.

Tokens
6.2K
Snippets
15
Records
31
Agent score
55%

What's inside parasail

  1. Introduction to parasail

    master

    parasail is a SIMD C (C99) library for pairwise sequence alignment. It implements Smith-Waterman (local), Needleman-Wunsch (global), and various semi-global algorithms (where insertions before the start or after the end of sequences are optionally not penalized).

    Supported algorithms include:

    • Diagonal [Wozniak, 1997]
    • Blocked [Rognes and Seeberg, 2000]
    • Striped [Farrar, 2007]
    • Prefix scan [Daily, 2015]

    Implementation Variants: To maintain high performance, parasail provides three distinct implementation variants. You should choose based on your performance/data requirements:

    1. Score only: Returns the alignment score and ending locations.
    2. Score + Statistics: Returns score, ending locations, and alignment statistics (exact matches, similarities, and alignment length).
    3. Traceback: Returns score, ending locations, and stores a traceback for later retrieval as a SAM CIGAR string.

    Note: When algorithms open a gap, only the gap open penalty is applied.

  2. Optimize database searches with Query Profiles

    master

    For _striped or _scan vector implementations, you can create a query profile to reuse the query sequence across multiple alignments. This significantly speeds up database searches where the query remains constant.

    Workflow:

    1. Create a profile using parasail_profile_create_{width} (e.g., parasail_profile_create_8).
    2. Use the profile with SSW-emulation functions like parasail_ssw_profile().
    3. Free the profile using parasail_profile_free(profile).

    Note: Profiles are opaque pointers; do not attempt to access their members directly.

    // Create a profile for 8-bit striped/scan implementations
    parasail_profile_t *profile = parasail_profile_create_8(s1, s1Len, matrix);
    
    // Use the profile for subsequent alignments
    parasail_result_ssw_t *result = parasail_ssw_profile(profile, s2, s2Len, 11, 1);
    
    // Cleanup
    parasail_result_ssw_free(result);
    parasail_profile_free(profile);
  3. Use the Enhanced Suffix Array Filter

    master

    The aligner uses a suffix array filter to skip sequence pairs that lack sufficient similarity.

    • Mechanism: It keeps only pairs containing an exact match of length $\ge$ cutoff (-c). This is a 'seed and extend' style approach, but once a match is found, the full alignment is performed rather than just extending the seed.
    • Default: The filter is enabled by default with a default cutoff of 7.
    • Disable: Use the -x flag to disable the filter.
  4. SIMDe API Naming and Type Conventions

    master

    SIMDe symbols are prefixed with simde_.

    Naming Examples:

    • The MMX intrinsic _mm_add_pi8 becomes simde_mm_add_pi8.
    • The type __m64 becomes simde__m64.

    Type Safety: To ensure portability, SIMDe often replaces standard types with fixed-width types. For example, where Intel APIs assume int is 32 bits, SIMDe uses int32_t (e.g., simde_mm_set_pi32 uses int32_t arguments). On platforms where the native API's assumptions already hold, these types remain compatible with existing code.

  5. Manage memory and batching in parasail_aligner

    master

    To prevent high memory usage during large alignments or traceback-capable routines, use batching to process alignments in chunks and free memory periodically.

    • Batch Size (-b): Specify the number of alignments to perform before writing output and freeing memory. Default is 0 (uses memory budget).
    • Memory Budget (-r): Specify the memory limit (e.g., 2GB). If not specified, the tool defaults to half of the system's available physical memory.

    Performance Tip: Larger batch sizes generally improve runtime performance but increase memory consumption. Tuning these allows you to balance speed vs. memory constraints.

  6. Understand parasail function naming conventions

    master

    Parasail contains over 1,000 functions. They are named using a structured pattern based on the algorithm, optimization, and data type.

    1. Algorithm Base:

    • sw: Smith-Waterman (local)
    • nw: Needleman-Wunsch (global)
    • sg: Semi-Global (no penalties at start/end of both sequences)
    • sg_qb, sg_qe, sg_qx, etc.: Various semi-global modes (e.g., qb = no penalty for gaps at beginning of query).

    2. Suffixes for Features:

    • _stats: Returns alignment statistics.
    • _table / _rowcol: Returns DP table or last row/column.
    • _scan: Uses prefix scan implementation.
    • _trace: Enables traceback capabilities.

    3. Vectorization & Optimization:

    • Non-vectorized: parasail_{algorithm}
    • Vectorized: parasail_{algorithm}_{strategy}_{instruction_set}_{width}
      • Strategies: _striped, _scan, _diag.
      • Instruction Sets: _sse2_128, _sse41_128, _avx2_256, _altivec_128, _neon_128.
      • Widths: _8, _16, _32, _64, _sat (attempts 8-bit, falls back to 16-bit if overflow occurs).

    4. Dispatchers: If you omit the instruction set and width, the function acts as a CPU dispatcher. It automatically selects the best available instruction set for the current CPU. This is the recommended way to call functions to ensure portability and performance.

    // Example: Needleman-Wunsch, stats, striped, SSE4.1, 16-bit
    parasail_nw_stats_striped_sse41_128_16
    
    // Example: Smith-Waterman, local, AVX2, 8-bit
    parasail_sw_scan_avx2_256_8
    
    // Example: Dispatcher (best instruction set automatically chosen)
    parasail_nw_stats_scan_16
  7. How parasail handles CPU dispatching

    master

    parasail uses runtime CPU dispatching to select the most efficient instruction set supported by the host CPU. This allows you to compile the library with support for advanced instruction sets even if your build machine doesn't support them, ensuring the resulting binary remains portable to older CPUs.

    Supported instruction sets:

    • SSE2
    • SSE4.1
    • AVX2
    • AltiVec
    • NEON
  8. Build parasail on Windows

    master

    Windows is fully supported. The preferred method for building on Windows is using CMake.

    1. Use the CMake GUI application to configure the build.
    2. You can configure the build for Visual Studio 2012 or newer (Visual Studio 2010 is supported but does not support AVX).
    3. Both 32-bit and 64-bit builds are supported.
  9. Install parasail using Meson

    master
    A contributed Meson build is available. Note that this build currently only supports the SSE and AVX Instruction Set Architectures (ISAs). Please refer to the Meson quick guide for standard usage instructions.
  10. Optimize SIMDe performance with OpenMP SIMD

    master

    For best performance, you should enable OpenMP 4 SIMD support.

    1. Define SIMDE_ENABLE_OPENMP before including any SIMDe headers.
    2. Enable OpenMP SIMD support in your compiler.

    Note: You do not need the full OpenMP runtime library; you can use compiler flags that enable only the SIMD support:

    • GCC: -fopenmp-simd
    • ICC: -openmp-simd
    #define SIMDE_ENABLE_OPENMP
    #include <x86/sse.h>

    Compilation (GCC)

    gcc -fopenmp-simd my_code.c -o my_code
  11. Cross-compile parasail for ARM

    master

    You can cross-compile for ARM platforms (aarch64, gnueabi, or gnueabihf) from an x86_64 Ubuntu host using the autotools build.

    1. Install Prerequisites:

    sudo apt-get install \
      gcc-arm-linux-gnueabi \
      g++-arm-linux-gnueabi \
      gcc-arm-linux-gnueabihf \
      g++-arm-linux-gnueabihf \
      gcc-aarch64-linux-gnu \
      g++-aarch64-linux-gnu

    2. Configure and Build: Use the --host and --build options. If you have zlib installed, you must use --without-zlib because the standard Ubuntu cross-compilation packages do not provide it.

    3. Verify with QEMU: After building, use QEMU to run the binaries on your x86_64 host. You must specify the QEMU_LD_PREFIX using the -L flag.

    TargetConfigure CommandQEMU Command
    aarch64./configure CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ --build x86_64-pc-linux-gnu --host aarch64-linux-gnu LDFLAGS=-static --without-zlibqemu-aarch64 -L /usr/aarch64-linux-gnu ./apps/parasail_aligner
    gnueabi./configure CC=arm-linux-gnueabi-gcc CXX=arm-linux-gnueabi-g++ --build x86_64-pc-linux-gnu --host arm-linux-gnueabi LDFLAGS=-static --without-zlibqemu-arm -L /usr/arm-linux-gnueabi ./apps/parasail_aligner
    gnueabihf./configure CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ --build x86_64-pc-linux-gnu --host arm-linux-gnueabihf LDFLAGS=-static --without-zlibqemu-arm -L /usr/arm-linux-gnueabihf ./apps/parasail_aligner
    # Example for aarch64
    ./configure CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ --build x86_64-pc-linux-gnu --host aarch64-linux-gnu LDFLAGS=-static --without-zlib
    make
    
    # Verify
    qemu-aarch64 -L /usr/aarch64-linux-gnu ./apps/parasail_aligner