DOCSEARCHbyAlgolia
Submit Docs

libmdbx Documentation

repository·master·Indexed 23 days ago

https://github.com/mithril-mine/libmdbx

A high-performance, embedded, transactional key-value database designed for multi-threaded process environments. A descendant of LMDB, libmdbx uses memory-mapping and B+trees to provide ACID compliance, non-blocking parallel reads, and serialized writes. It features a C API and a recommended modern C++ API for safer resource management. Distributed as amalgamated source code, it supports a wide range of platforms including Linux, Windows, macOS, and various POSIX-compliant systems.

Tokens
3.4K
Snippets
6
Records
25
Agent score
31%

What's inside libmdbx

  1. Overview of libmdbx

    master

    libmdbx is an extremely fast, compact, and powerful embedded transactional key-value database licensed under Apache 2.0. It is designed as a lightweight solution that allows a swarm of multi-threaded processes to ACIDly read and update several key-value maps and multimaps in a locally-shared database.

    Key properties include:

    • Extraordinary Performance: Uses Memory-Mapping and B+tree structures for $O(\log N)$ operation costs.
    • No Maintenance/Recovery: Does not use Write-Ahead Logging (WAL), meaning no crash recovery is required, though this is a consideration for write-intensive workloads requiring high durability.
    • Concurrency: Enforces serializability for writers via a single mutex and provides wait-free parallel reading without atomic/interlocked operations. Reading and writing transactions do not block each other.
    • Data Integrity: Guarantees data integrity after a crash unless durability was explicitly traded off for write performance.
    • Portability: Supports Linux, Windows, MacOS, Harmony, Android, iOS, FreeBSD, DragonFly, Solaris, OpenSolaris, OpenIndiana, NetBSD, OpenBSD, and other POSIX.1-2008 compliant systems.
    • Embedded Nature: Distributed as a few flat source files with no internal threads or server processes. It implements the Berkeley DB API with powerful extensions.

  2. Comparison: libmdbx vs LMDB

    master

    While libmdbx is compatible with many LMDB use cases, it offers several improvements:

    • Larger Keys: Supports keys significantly larger than LMDB's 511-byte limit (up to $\approx$ half the page size).
    • Performance: Generally up to 30% faster in CRUD benchmarks.
    • Reliability: Features like automatic size adjustment and continuous compactification are built-in, whereas LMDB may require manual intervention.
    • Features: Includes advanced features like transaction parking, 'get-cached' acceleration, and support for zero-length keys/values.

  3. Understand the libmdbx and MithrilDB roadmap

    master

    libmdbx is currently transitioning its distribution model. Starting from the end of 2025, libmdbx is distributed as amalgamated source code to simplify distribution and development.

    MithrilDB is the successor project currently under non-public development. Unlike libmdbx, MithrilDB will feature:

    • A new database format.
    • An API based on C++20.
    • Solutions to fundamental architectural problems found in libmdbx/LMDB.

    Note that while libmdbx will remain open-source and free, MithrilDB's availability and licensing may be subject to different restrictions.

  5. Automatic database size adjustment and compactification

    master

    libmdbx manages its own storage footprint through two main mechanisms:

    • On-the-fly size adjustment: The database automatically grows or shrinks based on parameters set via mdbx_env_set_geometry(). This includes defining the growth step and the truncation threshold.
    • Continuous compactification: During every commit, libmdbx automatically merges freeing pages that are adjacent to the unallocated area at the end of the file and truncates unused space. This provides zero-overhead database compactification.

    Note: Automatic size adjustment is not supported under Wine due to internal limitations, which will result in an MDBX_UNABLE_EXTEND_MAPSIZE error.

  6. Managing long-lived read transactions

    master

    Because libmdbx uses Copy-on-Write (CoW) for snapshot isolation, long-lived read transactions prevent the recycling of old/freed pages. If data is altered frequently during a long-running read, the database file can grow rapidly, potentially exhausting free space and degrading performance.

    To mitigate this, use one of the following strategies:

    1. Transaction Parking: Use the transaction parking mechanism to manage reader lifecycle.
    2. Handle-Slow-Readers callback: Implement a callback to handle or resolve slow readers.
    3. Avoid long-running reads: The best practice is to keep read transactions as short as possible.

  7. Configure libmdbx for cross-container interoperability

    master

    When running libmdbx in multiple containers or between a host and a container, three conditions must be met:

    1. Memory Mapping Coherence: The OS kernel must ensure a unified page cache so there is only a single physical copy of each memory-mapped DB page in system memory.
    2. PID Uniqueness/Visibility:
      • POSIX: PIDs must be unique across all processes operating on the DB. In Docker, use --pid=host or --pid=container:<name|id>.
      • Non-POSIX (Windows): Processes must have inter-visibility of handles. OpenProcess(SYNCHRONIZE, ..., PID) should return ERROR_ACCESS_DENIED for invalid PIDs, not ERROR_INVALID_PARAMETER.
    3. Library Compatibility: The versions of libmdbx and libc/pthreads (e.g., glibc, musl) must be compatible. The options: string in the output of mdbx_chk -V must match between the host and containers. Avoid mixing different libc implementations (e.g., mixing glibc with musl).

  8. Understand libmdbx data integrity modes

    master

    libmdbx provides different levels of synchronization and safety for data durability:

    • MDBX_SAFE_NOSYNC: A proposed trade-off that uses an append-like manner for updates. This avoids database corruption after a system crash, even in asynchronous unordered write-to-disk mode.
    • MDBX_UTTERLY_NOSYNC: Matches the behavior of LMDB's MDB_NOSYNC for users who require that specific performance/risk profile.

  9. Choose a write mode for libmdbx

    master

    libmdbx provides three distinct write modes that allow you to balance data durability against write performance. The choice depends on your application's tolerance for data loss in the event of a system crash.

    Sync-write mode

    • Durability: Highest. In case of a crash, all data is consistent and conforms to the last successful transaction.
    • Mechanism: Uses the fdatasync syscall after each write transaction.
    • Use Case: Applications requiring strict ACID compliance where every transaction must be safely persisted to disk immediately.

    Lazy-write mode

    • Durability: Medium. In case of a crash, data is consistent up to the last successful transaction, but subsequent transactions may be lost. Unlike other engines, libmdbx does not use a Write-Ahead Log (WAL) or transaction journal; it relies on the filesystem and OS kernel (mmap) to handle I/O.
    • Use Case: Applications that can tolerate some data loss in exchange for significantly higher throughput.

    Async-write mode

    • Durability: Lowest. In case of a crash, data is consistent up to the last successful transaction, but the number of lost transactions is much higher than in lazy-write mode.
    • Mechanism: Uses msync(MS_ASYNC) to perform as few writes as possible to persistent storage.
    • Use Case: High-performance scenarios where maximum throughput is required and data loss is acceptable.

  10. How libmdbx handles concurrency and writes

    master

    libmdbx uses a Multi-Version Concurrency Control (MVCC) model with Copy-on-Write (CoW) and shadow paging. This provides several concurrency guarantees:

    • Non-blocking Readers: Readers do not block writers, and writers do not block readers. Reads scale linearly across CPUs.
    • Serialized Writes: There can be only one writer at a time. libmdbx does not support multiple concurrent write transactions. This design prevents transaction conflicts and deadlocks.
    • No WAL/Journal: Unlike many databases, libmdbx does not use a Write-Ahead Log (WAL) or a transaction journal. It uses shadow paging, meaning no crash recovery is required and no maintenance is needed, but syncing data to disk can become a bottleneck for write-intensive workloads.

  11. Build libmdbx on Windows

    master

    For Windows, the recommended approach is using CMake and Microsoft Visual Studio 2019 (or newer) with a recent Windows SDK to ensure proper C11 and alignas() support.

    MinGW: Requires version 10.2 or recent, coupled with a modern CMake.

    Reducing Runtime Dependencies: To avoid runtime dependencies from the MSVC CRT and other MSVC libraries, use the following CMake option:

    -DMDBX_WITHOUT_MSVC_CRT:BOOL=ON

    Note: If using non-standard build methods, ensure ntdll.lib is added to the linking stage.

    -DMDBX_WITHOUT_MSVC_CRT:BOOL=ON

  12. Achieve reproducible builds with MDBX_BUILD_TIMESTAMP

    master

    By default, libmdbx tracks the build time via the MDBX_BUILD_TIMESTAMP build option and macro. To ensure reproducible builds, you must override this with a fixed string value.

    Using make:

    make MDBX_BUILD_TIMESTAMP=unknown

    Using cmake:

    cmake -DMDBX_BUILD_TIMESTAMP:STRING=unknown
    make MDBX_BUILD_TIMESTAMP=unknown
cmake -DMDBX_BUILD_TIMESTAMP:STRING=unknown