MMseqs2
repository·master·Indexed 24 days ago
https://github.com/soedinglab/mmseqs2An ultra-fast and sensitive software suite for searching and clustering massive protein and nucleotide sequence datasets, designed for high scalability across multiple cores and servers.
What's inside MMseqs2
- microtar is a lightweight tar library written in ANSI C. This specific implementation has been adapted from the original microtar to be read-only and to support fast seeking within tar archives.
Overview of Zstandard (zstd)
masterZstandard, orzstd, is a fast lossless compression algorithm designed for real-time compression scenarios. It aims to provide compression ratios comparable to or better thanzlibwhile maintaining high speeds. It is available as an open-source C library and a command-line utility that supports producing and decoding.zst,.gz,.xz, and.lz4files.Use the NibbleAndAHalf base64 library
masterNibbleAndAHalf is a fast ANSI C library for base64 encoding and decoding. It is distributed as a single-header library, making it easy to integrate into C projects.
To use the library in your project, you only need to include the main header file:
#include "base64.h".Note that all test-related functions have been moved to a separate header,
testbase64.h, so they are not included in the standardbase64.himport.#include "base64.h"Use SIMDe as a submodule without test cases
masterThis repository provides a lightweight version of the SIMDe library containing only the core functionality. It is designed to be used as a git submodule in projects that require SIMD emulation but want to avoid the overhead and size of the full SIMDe test suite.
Note: This is a generated repository. All development and issue reporting should be directed to the main SIMDe repository. Do not file issues or pull requests here.
Integrate Rust into CMake using Corrosion
masterCorrosion (formerly
cmake-cargo) is a tool designed to integrate Rust code into existing CMake projects. It automates the process of importing Rust-generated executables, static libraries, and dynamic libraries into the CMake build system by reading theCargo.tomlmanifest.Key capabilities include:
- Automatic import of executables, static, and shared libraries from Rust crates.
- Easy installation of Rust executables.
- Trivial linking of Rust executables to C/C++ libraries within the project tree.
- Support for Multi-Config generators and simple cross-compilation.
Use the largeNbDicts benchmark tool
masterlargeNbDictsis a benchmark tool designed to test dictionary decompression performance in scenarios where a very large number of dictionaries are used. It specifically targets the 'cold' dictionary scenario, where frequent dictionary changes cause increased latency due to cache misses. The tool allows users to investigate performance and experiment with mitigation techniques for this specific workload.largeNbDicts [Options] filename(s)What is Corrosion and how does it integrate Rust with CMake
masterCorrosion (formerly
cmake-cargo) is a tool designed to integrate Rust into existing CMake projects. It automatically imports Rust executables, static libraries, and dynamic libraries from a Rust package or workspace as native CMake targets.Key integration features:
- Libraries: Imported static and dynamic libraries can be linked into C/C++ CMake targets using standard CMake commands like
target_link_libraries(). - Rust Targets: For Rust executables and dynamic libraries, Corrosion provides a helper function
corrosion_link_librariesto simplify adding the necessary flags required to link C/C++ libraries into the Rust target.
- Libraries: Imported static and dynamic libraries can be linked into C/C++ CMake targets using standard CMake commands like
Zstandard frame types: Zstandard frames vs Skippable frames
masterZstandard defines two distinct frame formats:
- Zstandard frames: These are the standard frames used to store compressed data.
- Skippable frames: These are specialized frames used to store custom user metadata. They are designed to be skippable by decoders that do not recognize the metadata, allowing the stream to continue to the next valid Zstandard frame.
Understand the Zstandard compression format specification
masterThe Zstandard compression format is defined inzstd_compression_format.md. Developers implementing compliant encoders or decoders must adhere to the specifications laid out in this document to ensure interoperability.Understand Dictionary Builder benchmarking results
masterThe benchmarking output follows a specific structure for comparing optimized vs. non-optimized parameters:
- Cover Comparison: The first
Covervalue represents the optimized cover. The secondCovervalue uses the optimizeddandkparameters derived from the first one. - fastCover Comparison: For every
fvalue offastCover, the first entry is the optimizedfastCover. The second entry uses the optimizeddandkfrom the first entry. This is evaluated foraccelvalues ranging from 1 to 10. - Parameter Columns: In the output, the fourth column represents the chosen
dvalue, and the fifth column represents the chosenkvalue.
- Cover Comparison: The first
Convert weights to Huffman prefix codes
masterTo transform a list of decoded weights into Huffman prefix codes, follow these steps:
- Calculate Bits: For each symbol, calculate
Number_of_Bits = (Weight > 0) ? Max_Number_of_Bits + 1 - Weight : 0. - Sort: Sort symbols by
Weight. If weights are equal, maintain their natural sequential order. - Filter: Remove symbols with a
Weightof zero. - Distribute: Starting from the lowest
Weight, distribute prefix codes in sequential order.
Example Transformation:
Literal 0 1 2 3 4 5 Weight4 3 2 0 1 1 Sorted Distribution:
Literal 3 4 5 2 1 0 Weight0 1 1 2 3 4 Number_of_Bits0 4 4 3 2 1 prefix codes N/A 0000 0001 001 01 1 - Calculate Bits: For each symbol, calculate
Understand Block structure and types
masterAfter the header, a frame contains one or more blocks. Each block consists of a 3-byte header and the block content.
Block Header (3 bytes, little-endian):
Field Bits Description Last_Blockbit 0 If set, this is the final block in the frame. Block_Typebits 1-2 The type of block (see below). Block_Sizebits 3-23 Size of Block_Contentin bytes.Block Types:
Raw_Block(0): Uncompressed data.Block_Contentis the data itself.RLE_Block(1): Run-Length Encoding.Block_Contentis a single byte to be repeatedBlock_Sizetimes.Compressed_Block(2): Contains Zstandard compressed data (Literals and Sequences).Reserved(3): Invalid/Corrupted data.
Block Size Constraint:
Block_Sizeis the smallest ofWindow_Sizeor 128 KB. ForCompressed_Block,Block_Sizemust be strictly less than the decompressed size.