MMseqs2

repository·master·Indexed 24 days ago

https://github.com/soedinglab/mmseqs2

An ultra-fast and sensitive software suite for searching and clustering massive protein and nucleotide sequence datasets, designed for high scalability across multiple cores and servers.

Tokens
38.6K
Snippets
90
Records
232
Agent score
84%

What's inside MMseqs2

  1. Overview of Zstandard (zstd)

    master
    Zstandard, or zstd, is a fast lossless compression algorithm designed for real-time compression scenarios. It aims to provide compression ratios comparable to or better than zlib while maintaining high speeds. It is available as an open-source C library and a command-line utility that supports producing and decoding .zst, .gz, .xz, and .lz4 files.
  2. Use the NibbleAndAHalf base64 library

    master

    NibbleAndAHalf is a fast ANSI C library for base64 encoding and decoding. It is distributed as a single-header library, making it easy to integrate into C projects.

    To use the library in your project, you only need to include the main header file: #include "base64.h".

    Note that all test-related functions have been moved to a separate header, testbase64.h, so they are not included in the standard base64.h import.

    #include "base64.h"
  3. Use SIMDe as a submodule without test cases

    master

    This repository provides a lightweight version of the SIMDe library containing only the core functionality. It is designed to be used as a git submodule in projects that require SIMD emulation but want to avoid the overhead and size of the full SIMDe test suite.

    Note: This is a generated repository. All development and issue reporting should be directed to the main SIMDe repository. Do not file issues or pull requests here.

  4. Integrate Rust into CMake using Corrosion

    master

    Corrosion (formerly cmake-cargo) is a tool designed to integrate Rust code into existing CMake projects. It automates the process of importing Rust-generated executables, static libraries, and dynamic libraries into the CMake build system by reading the Cargo.toml manifest.

    Key capabilities include:

    • Automatic import of executables, static, and shared libraries from Rust crates.
    • Easy installation of Rust executables.
    • Trivial linking of Rust executables to C/C++ libraries within the project tree.
    • Support for Multi-Config generators and simple cross-compilation.
  5. Use the largeNbDicts benchmark tool

    master

    largeNbDicts is a benchmark tool designed to test dictionary decompression performance in scenarios where a very large number of dictionaries are used. It specifically targets the 'cold' dictionary scenario, where frequent dictionary changes cause increased latency due to cache misses. The tool allows users to investigate performance and experiment with mitigation techniques for this specific workload.

    largeNbDicts [Options] filename(s)
  6. What is Corrosion and how does it integrate Rust with CMake

    master

    Corrosion (formerly cmake-cargo) is a tool designed to integrate Rust into existing CMake projects. It automatically imports Rust executables, static libraries, and dynamic libraries from a Rust package or workspace as native CMake targets.

    Key integration features:

    • Libraries: Imported static and dynamic libraries can be linked into C/C++ CMake targets using standard CMake commands like target_link_libraries().
    • Rust Targets: For Rust executables and dynamic libraries, Corrosion provides a helper function corrosion_link_libraries to simplify adding the necessary flags required to link C/C++ libraries into the Rust target.
  7. Zstandard frame types: Zstandard frames vs Skippable frames

    master

    Zstandard defines two distinct frame formats:

    1. Zstandard frames: These are the standard frames used to store compressed data.
    2. Skippable frames: These are specialized frames used to store custom user metadata. They are designed to be skippable by decoders that do not recognize the metadata, allowing the stream to continue to the next valid Zstandard frame.
  8. Understand Dictionary Builder benchmarking results

    master

    The benchmarking output follows a specific structure for comparing optimized vs. non-optimized parameters:

    • Cover Comparison: The first Cover value represents the optimized cover. The second Cover value uses the optimized d and k parameters derived from the first one.
    • fastCover Comparison: For every f value of fastCover, the first entry is the optimized fastCover. The second entry uses the optimized d and k from the first entry. This is evaluated for accel values ranging from 1 to 10.
    • Parameter Columns: In the output, the fourth column represents the chosen d value, and the fifth column represents the chosen k value.
  9. Convert weights to Huffman prefix codes

    master

    To transform a list of decoded weights into Huffman prefix codes, follow these steps:

    1. Calculate Bits: For each symbol, calculate Number_of_Bits = (Weight > 0) ? Max_Number_of_Bits + 1 - Weight : 0.
    2. Sort: Sort symbols by Weight. If weights are equal, maintain their natural sequential order.
    3. Filter: Remove symbols with a Weight of zero.
    4. Distribute: Starting from the lowest Weight, distribute prefix codes in sequential order.

    Example Transformation:

    Literal012345
    Weight432011

    Sorted Distribution:

    Literal345210
    Weight011234
    Number_of_Bits044321
    prefix codesN/A00000001001011
  10. Understand Block structure and types

    master

    After the header, a frame contains one or more blocks. Each block consists of a 3-byte header and the block content.

    Block Header (3 bytes, little-endian):

    FieldBitsDescription
    Last_Blockbit 0If set, this is the final block in the frame.
    Block_Typebits 1-2The type of block (see below).
    Block_Sizebits 3-23Size of Block_Content in bytes.

    Block Types:

    • Raw_Block (0): Uncompressed data. Block_Content is the data itself.
    • RLE_Block (1): Run-Length Encoding. Block_Content is a single byte to be repeated Block_Size times.
    • Compressed_Block (2): Contains Zstandard compressed data (Literals and Sequences).
    • Reserved (3): Invalid/Corrupted data.

    Block Size Constraint: Block_Size is the smallest of Window_Size or 128 KB. For Compressed_Block, Block_Size must be strictly less than the decompressed size.