DOCSEARCHbyAlgolia
Submit Docs

LLFIO Documentation

repository·develop·Indexed 21 days ago

https://github.com/ned14/llfio

A high-performance, zero-copy C++ library for file I/O and filesystem operations optimized for ultra-low latency storage, SCM/DAX, and high-throughput environments. LLFIO v2 features a hierarchical handle system (including path_handle and mapped_file_handle), supports C++17 and C++20, and is compatible with Windows, Linux, Android, iOS, Mac OS, and FreeBSD. It is designed to minimize software overhead, achieving benchmarked I/O overhead of ~100 nanoseconds on Linux.

Tokens
2.8K
Snippets
5
Records
16
Agent score
77%

What's inside LLFIO

  1. Overview of LLFIO

    develop

    LLFIO is a high-performance, zero-copy file I/O and filesystem library designed for low-latency storage devices (e.g., ~1 microsecond 4Kb transfer latencies) and Storage Class Memory (SCM)/Direct Access Storage (DAX).

    Key performance characteristics:

    • Benchmarked I/O overhead of ~100 nanoseconds on Linux.
    • Theoretical maximum of 10M IOPS @ QD1 and ~40GB/sec per thread.
    • Zero malloc, zero exception throwing, and zero whole-system memory copies (including paths).
    • Race-free filesystem design (no TOCTOU).
    • Supports C++17 and C++20 (utilizing Coroutines, Concepts, Span, and Byte if available).
    • Compatible with Windows, Linux, Android, iOS, Mac OS, and FreeBSD.

  2. Overview of the exploratory ACID key-value store

    develop

    This project is an exploratory, toy ACID key-value store built using the LLFIO library. It is designed to test the feasibility of specific implementation approaches and to evaluate the LLFIO design.

    Warning: This is a toy implementation. It should not be used for any production or serious applications.

    Key Features:

    • Lookup: Retrieve any BLOB value using a 128-bit key.
    • Atomic Transactions: Update up to 65,535 key-value pairs in a single atomic transaction.

  3. Platform support and feature availability in LLFIO v2

    develop

    LLFIO v2 provides high-performance file I/O abstractions with varying support across Windows and POSIX platforms. Key features include:

    Cross-Platform (Windows & POSIX)

    • Handle Management: Native handle cloning.
    • Caching: Up to seven forms of kernel caching.
    • I/O Operations: Synchronous and Asynchronous universal scatter-gather I/O; I/O deadlines and cancellation.
    • Locking: shared_fs_mutex supporting shared/exclusive locking via lock files, byte ranges, atomic appends, memory maps, and safe byte ranges.
    • Path Handling: Universal portable UTF-8 path views.
    • Algorithms: Graph-based directory hierarchy traversal, summary, and reliable deletion; intelligent file contents cloning.
    • Memory: Large, huge, and massive page size support; llfio::algorithm::trivial_vector<T> with constant time reallocation for trivially copyable types.

    Windows Specific

    • Path Support: Win32 path support (260 limit) and NT kernel path support (32,768 limit).
    • File Operations: Absolute path open; relative "anchored" path open (race-free); retrieving/setting maximum file extent; retrieving current path after renames; hole punching/enumeration; directory handles; symlink handles.

    POSIX Specific

    • Memory: (Limited) Large/huge page support for file maps.

    Note: Features like directory change monitoring and ACL permissions support are planned for future releases.

  4. LLFIO Handle Type Hierarchy

    develop

    LLFIO v2 uses a hierarchical handle system to provide specialized capabilities. Instead of a generic void*, it uses a native handle/fd abstraction that evolves through the following types:

    • handle: Base type providing open, close, get path, clone, set/unset append only, change caching, and characteristics.
    • fs_handle: Handles that include an inode number.
    • path_handle: A race-free anchor to a subset of the filesystem.
    • directory_handle: Used for enumerating the filesystem.
    • io_handle: Adds synchronous scatter-gather I/O and byte range locking.
    • file_handle: Adds open/create file operations and get/set maximum extent.
    • mapped_file_handle: Adds low-latency memory-mapped scatter-gather I/O.

  5. Performance characteristics of LLFIO

    develop

    LLFIO is designed to minimize software overhead to approach the raw hardware latency of storage devices. It achieves near-native performance for direct transfers (QD1 and QD4) across various media types.

    Direct Transfer Latencies (QD1 4Kb)

    LLFIO aims to keep software latency below 99% of the physical hardware latency:

    • NVMe Flash: Windows (~37us), FreeBSD (~70us), Linux (~30us).
    • SATA Flash: Windows (~290us), Linux (~158us).
    • Spinning Rust: Windows (~187,231us), FreeBSD (~9,836us), Linux (~26,484us).

    Direct Transfer Latencies (QD4 4Kb - 75% Read / 25% Write)

    • NVMe Flash: Windows (~50us), FreeBSD (~143us), Linux (~40us).
    • SATA Flash: Windows (~1,812us), Linux (~1,416us).
    • Spinning Rust: Windows (~48,185us), FreeBSD (~61,834us), Linux (~104,507us).

  6. Warning: Header-only mode on Windows

    develop

    Using LLFIO_HEADERS_ONLY=1 on Microsoft Windows is considered unsafe for anything beyond toy projects due to design flaws in <system_error> regarding custom error code categories in shared libraries. This can lead to unreliable semantic comparisons of error codes.

    Recommended solutions for Windows:

    1. Use the experimental SG14 status_code by defining LLFIO_EXPERIMENTAL_STATUS_CODE=1.
    2. Use the NT kernel error category as a shared library.
    3. Avoid header-only mode and use static or shared library builds instead.

  7. Clone the LLFIO source repository

    develop

    When cloning on Windows, it is recommended to enable long paths to ensure deep directory structures in submodules do not cause errors. Use the --recursive flag to ensure all submodules are included.

    # Enable long paths (may require elevated privileges)
git config --system core.longpaths true

# Clone with submodules
git clone --recursive https://github.com/ned14/llfio.git

# If you already cloned without --recursive, run this inside the directory:
git submodule update --init --recursive

  8. Verify repository commits and tags using PGP

    develop

    To ensure the integrity and authenticity of the source code, commits and tags in the ned14/llfio repository can be verified using the provided PGP public key. Use a GnuPG compatible tool to import the key and verify signatures.

    -----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2

mDMEVvMacRYJKwYBBAHaRw8BAQdAp+Qn6djfxWQYtAEvDmv4feVmGALEQH/pYpBC
llaXNQe0WE5pYWxsIERvdWdsYXMgKHMgW3VuZGVyc2NvcmVdIHNvdXJjZWZvcmdl
IHthdH0gbmVkcHJvZCBbZG90XSBjb20pIDxzcGFtdHJhcEBuZWRwcm9kLmNvbT6I
eQQTFggAIQUCVvMacQIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRCELDV4
Zvkgx4vwAP9gxeQUsp7ARMFGxfbR0xPf6fRbH+miMUg2e7rYNuHtLQD9EUoR32We
V8SjvX4r/deKniWctvCi5JccgfUwXkVzFAk=
=puFk
-----END PGP PUBLIC KEY BLOCK-----

  9. Use the single-header editions of AFIO

    develop
    AFIO provides two single-header editions designed to minimize project footprint. To maintain a smaller file size, these editions are platform-specific and are split into separate files for POSIX and Windows environments. Choose the header that matches your target operating system.

  10. Build the AFIO documentation

    develop

    To build the AFIO (Boost.AFIO) documentation, you must set up a specific toolchain involving Doxygen, Python, and DocBook XSLTs. The process involves generating QuickBook files from Doxygen XML and then using Boost's build system (b2) to render them into HTML or PDF.

    Prerequisites

    1. Install doxygen, java, and python 2.7 and ensure they are in your PATH.
    2. Create a tool directory (e.g., C:\BoostBook or /home/ned/BoostBook) that contains no spaces.
    3. Install xsltproc, libxml2, libxslt, and the Docbook XSL -1 version (not the -ns version) into your tool directory.
      • Note: Use the -1 version from SourceForge (e.g., version 1.78.1).
    4. RenderX XEP: Because Apache FOP has compatibility issues with current DocBook XSLTs, use RenderX XEP instead. Configure your user-config.jam with the appropriate paths to use XEP.

    Build Steps

    1. Generate QuickBook files: Navigate to libs/afio/doc and set the DOXYGEN_XML2QBK environment variable to point to your doxygen_xml2qbk.exe binary, then run the python script:

      set DOXYGEN_XML2QBK="C:\BoostBook\bin\doxygen_xml2qbk.exe"
python make_qbk.py

    2. Build HTML documentation: Return to the Boost top directory, navigate to the doc directory, and run b2 targeting the AFIO doc path:

       cd doc
 ../b2 ../libs/afio/doc

    3. Build PDF documentation: From the Boost top directory, run b2 with the pdf target:

       ../b2 ../libs/afio/doc pdf
    # 1. Generate QuickBook files
set DOXYGEN_XML2QBK="C:\BoostBook\bin\doxygen_xml2qbk.exe"
python make_qbk.py

# 2. Build HTML
cd doc
../b2 ../libs/afio/doc

# 3. Build PDF
../b2 ../libs/afio/doc pdf

  11. Install LLFIO via vcpkg

    develop

    The easiest way to install LLFIO on macOS, Linux, or Microsoft Windows is using the vcpkg package manager. Once installed, you can include the library using the header <llfio/llfio.hpp>.

    vcpkg install llfio

  12. Build LLFIO static libraries from source

    develop

    Building static libraries requires CMake (v3.9 or better) and must be performed as an out-of-tree build.

    POSIX (Linux/macOS)

    mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . --parallel
ctest -R llfio_sl

    Windows/macOS (Visual Studio/Xcode)

    mkdir build
cd build
cmake .. -G<your generator here>
cmake --build . --parallel --config Release
ctest -C Release -R llfio_sl