DOCSEARCHbyAlgolia
Submit Docs

NuDB Documentation

repository·master·Indexed 19 days ago

https://github.com/cppalliance/nudb

NuDB is a high-performance, append-only, header-only C++ key/value store optimized for random read performance on SSDs and high-IOPS devices. Designed for content-addressable storage, it supports database sizes up to 281TB with fixed-size keys and values ranging from 1 byte to 4GB. It requires C++11 or greater and Boost 1.69+, linking against Boost.Thread and Boost.System.

Tokens
2K
Snippets
2
Records
10
Agent score
15%

What's inside NuDB

  1. Overview of NuDB

    master

    NuDB is an append-only, key/value store optimized for random read performance on modern SSDs or high-IOPS devices. It is designed for use cases like content-addressable storage where cryptographic digests serve as keys.

    Key characteristics:

    • Performance: Read performance and memory usage are independent of database size.
    • Capacity: Supports database sizes up to 281TB.
    • Data Model: All keys are the same size; values range from 1 to 4GB ($2^{32}$ bytes).
    • Operations: Append-only (no updates or deletes); inserts are atomic and consistent.
    • Architecture: Header-only library; optimized for concurrent fetch; key and data files can reside on different devices.

  2. Understand NuDB concurrency and consistency

    master

    NuDB is designed with the following concurrency and consistency models:

    • Concurrency: Fetches are fully concurrent and do not hold locks during disk I/O. Inserts are serialized to ensure atomicity and prevent duplicate keys.
    • Visibility: Insertions are buffered in memory and become immediately discoverable by subsequent or concurrent fetch calls. A dedicated thread periodically commits buffered data to disk (at least once per second or during high activity).
    • Atomicity: Inserts are atomic; they either succeed immediately or fail.
    • Recovery: NuDB uses a log file to store bookkeeping information. In the event of an external failure, the recovery process uses this log to roll back partial commits and restore consistency by applying known good bucket snapshots.

  3. Integrate NuDB into your project

    master

    NuDB is a header-only library. To integrate it:

    1. Copy the NuDB source files into your project's source tree (or use git submodule/git subtree).
    2. Add the include/ directory to your compiler's include search paths.
    3. Include the header in your code:
      #include <nudb/nudb.hpp>
    4. Linking: You must link against the Boost.Thread and Boost.System libraries.

  4. Build NuDB tests and examples with CMake

    master

    To generate build scripts for the included tests and examples using CMake, run the following commands from the repository root:

    For 32-bit Windows:

    cd bin
cmake ..

    For Linux/Mac or 64-bit Windows:

    cd ../bin64
cmake ..
# OR for 64-bit Windows with Visual Studio:
cmake -G"Visual Studio 14 2015 Win64" ..
    cd bin
cmake ..

cd ../bin64
cmake ..
# OR
cmake -G"Visual Studio 14 2015 Win64" ..

  5. Requirements for using NuDB

    master

    To use NuDB, your environment must meet the following requirements:

    Core Requirements:

    • Boost: Version 1.69 or higher.
    • C++ Standard: C++11 or greater.
    • Hardware: SSD drive or an equivalent high-IOPS device.

    Optional (for building tests and examples):

    • CMake: 3.7.2 or later.
    • Boost.Build (b2): Properly configured.

  6. Configure NuDB database creation parameters

    master

    When creating a new NuDB database, you must define the following parameters to establish the storage structure:

    • KeySize: The fixed size of a key in bytes.
    • BlockSize: The physical size of a key file record. This should ideally match the sector size or block size of your underlying physical media. While a default of 4096 is typical, functions are provided to estimate the best value for a specific device.
    • LoadFactor: The desired fraction of bucket occupancy (e.g., 0.50). This helps balance bucket occupancy against the likelihood of overflows.

  7. Configure NuDB database opening parameters

    master

    When opening an existing NuDB database, you must provide the following parameters:

    • Appnum: An application-defined integer constant used to identify the data format or context. This value is stored in the database and can be retrieved later.
    • AllocSize: A hint for the memory recycler representing a significant multiple of your average data size. For example, if your average data size is 1KB, an AllocSize of 16MB is recommended. Setting this too low will reduce the efficiency of the memory recycler.

  8. Basic NuDB usage example

    master

    This example demonstrates the full lifecycle of a NuDB instance: creating a database, opening it, inserting key/value pairs, fetching them, and closing the database.

    #include <nudb/nudb.hpp>
#include <cstddef>
#include <cstdint>

int main()
{
    using namespace nudb;
    std::size_t constexpr N = 1000;
    using key_type = std::uint32_t;
    error_code ec;
    auto const dat_path = "db.dat";
    auto const key_path = "db.key";
    auto const log_path = "db.log";
    create<xxhasher>(
        dat_path, key_path, log_path,
        1,
        make_salt(),
        sizeof(key_type),
        block_size("."),
        0.5f,
        ec);
    store db;
    db.open(dat_path, key_path, log_path, ec);
    char data = 0;
    // Insert
    for(key_type i = 0; i < N; ++i)
        db.insert(&i, &data, sizeof(data), ec);
    // Fetch
    for(key_type i = 0; i < N; ++i)
        db.fetch(&i,
            [&](void const* buffer, std::size_t size)
        {
            // do something with buffer, size
        }, ec);
    db.close(ec);
    erase_file(dat_path);
    erase_file(key_path);
    erase_file(log_path);
}

  9. Use fetch and insert operations

    master

    NuDB provides two primary operations for interacting with the key/value store:

    fetch

    Retrieves a variable-length value associated with a given key. The caller provides a factory to supply the buffer for storing the value, allowing for custom memory allocation strategies.

    insert

    Adds a key/value pair to the store.

    • Constraints: Value data must contain at least one byte. Duplicate keys are not allowed and will cause the operation to fail.
    • Concurrency: Inserts are serialized and atomic. Once an insert succeeds, the key is immediately visible to subsequent fetch calls.

  10. Reference the NuDB file formats

    master

    NuDB uses three primary files. All integer values are stored as big-endian. The uint48_t format uses 6 bytes.

    Key File

    Contains a Header followed by fixed-length Bucket Records.

    • Header (104 bytes): Includes Type ("nudb.key"), Version, UID, Appnum, KeySize, Salt, Pepper, BlockSize, LoadFactor, and Reserved space.
    • Bucket Record: Contains Count (number of keys), Spill (offset to next spill record or 0), and an array of BucketEntry items.
    • Bucket Entry: Contains Offset (in data file), Size (value size), and Hash (key hash).

    Data File

    Contains a Header followed by variable-length Value Records or Spill Records.

    • Header (92 bytes): Includes Type ("nudb.dat"), Version, UID, Appnum, KeySize, and Reserved space.
    • Data Record: Contains Size (value size), Key (the key itself), and Data (the value bytes).
    • Spill Record: A fixed-length record used when a bucket overflows. It contains a Zero identifier, Size (for skipping), and the SpillBucket (the actual Bucket Record).

    Log File

    Contains a Header followed by fixed-size log records used for recovery.

    • Header (62 bytes): Includes Type ("nudb.log"), Version, UID, Appnum, KeySize, Salt, Pepper, BlockSize, KeyFileSize, and DataFileSize.
    • Log Record: Contains a Index (0-based bucket index) and a Bucket (compacted bucket record containing only size entries).