DOCSEARCHbyAlgolia
Submit Docs

heed

repository·main·Indexed 21 days ago

https://github.com/meilisearch/heed

A fully typed LMDB (mdb.master) wrapper for Rust with minimum overhead. It provides high-level abstractions for storing Rust types, including Serde-compatible types via bincode, JSON, and MessagePack. The project includes the heed crate for standard LMDB and the heed3 crate for the mdb.master3 branch featuring encryption-at-rest. It offers specialized traits for byte encoding/decoding, custom key comparators, and lazy decoding to defer CPU-intensive operations.

Tokens
21.4K
Snippets
74
Records
100
Agent score
74%

What's inside heed

  2. Difference between heed and heed3 crates

    main

    The project provides two crates: heed and heed3.

    • heed: Supports the standard LMDB mdb.master branch.
    • heed3: Supports the mdb.master3 branch, which features encryption-at-rest.

    Both crates share a codebase. Developers working on heed3 can use the convert-to-heed3.sh script to move the heed3/Cargo.toml to the heed/ folder and update namespace references from heed:: to heed3:: automatically.

  3. Build heed from source

    main

    To build the project from source, clone the repository recursively to ensure submodules are included, then use cargo build.

    If you have already cloned the repository without initializing submodules, run git submodule update --init first.

    git clone --recursive https://github.com/meilisearch/heed.git
cd heed
cargo build

  4. Use heed-types for database serialization

    main
    The heed-types crate provides specialized types designed for efficient serialization and deserialization of data stored within databases. It includes support for various formats such as JSON, Bincode, and MessagePack via feature flags, and provides primitives for handling bytes, strings, integers, and lazy decoding.

  5. Configure read-only iteration methods

    main

    When using a read-only iterator (RoIter), you can choose how to handle duplicate values (if the database uses DUP_SORT).

    • move_between_keys(): Moves to the next unique key, ignoring duplicate values for the same key.
    • move_through_duplicate_values(): Iterates through every single key/value entry, including duplicates.

    Both methods return a new iterator with the specified IterationMethod.

    // Example: Moving between unique keys
let mut iter = db.iter(&wtxn)?.move_between_keys();

// Example: Iterating through all duplicates
let mut iter = db.iter(&wtxn)?.move_through_duplicate_values();

  6. Use EncryptedEnv for encrypted LMDB environments

    main

    An EncryptedEnv is an environment handle constructed using EnvOpenOptions::open_encrypted. It provides an interface for managing databases within an encrypted LMDB environment. It supports standard operations like creating transactions, opening databases, and managing environment flags, while ensuring data is encrypted at rest.

    Key capabilities include:

    • Transactions: Create read-only (read_txn, static_read_txn) and read-write (write_txn, nested_write_txn) transactions.
    • Database Management: Open existing databases (open_database) or create new ones (create_database).
    • Environment Control: Manage flags via set_flags, resize the memory map via resize, and perform backups via copy_to_file.
    // Example of opening an encrypted environment
let mut key = Key::default();
// ... derive key from password ...

let env = unsafe {
    let mut options = EnvOpenOptions::new().read_txn_without_tls();
    options
        .map_size(2 * 1024 * 1024 * 1024) // 2 GiB
        .open_encrypted::<ChaCha20Poly1305, _>(key, &env_path)?
};

  7. Difference between heed and heed3

    main

    The heed repository provides two distinct wrappers for LMDB:

    • heed: A wrapper around the mdb.master branch of LMDB.
    • heed3: A wrapper around the mdb.master3 branch. It is intended for use when you need encryption-at-rest and checksumming features, which are not available in the standard heed crate. heed3 will be considered stable once the underlying LMDB branch is officially released.

  8. Configure prefix iteration behavior

    main

    When using prefix iterators (RoPrefix, RwPrefix, RoRevPrefix, RwRevPrefix), you can control how duplicates are handled using the following methods:

    • move_between_keys(): Configures the iterator to move to the next unique key, effectively ignoring duplicate values for the same key.
    • move_through_duplicate_values(): Configures the iterator to yield every single key/value entry, including duplicates.

  9. Configure iteration behavior for range iterators

    main

    Range iterators in heed allow you to control how duplicate keys are handled during iteration using two primary methods:

    1. move_between_keys(): Moves the cursor to the next unique key, effectively ignoring duplicate values for the same key.
    2. move_through_duplicate_values(): Moves through every single entry, including those where the key is identical to the previous one.

    These methods return a new iterator instance with the specified IterationMethod (either MoveBetweenKeys or MoveThroughDuplicateValues).

  10. Iterate in reverse order

    main

    Heed provides reverse iterators for both read-only and read-write operations:

    • RoRevIter: A read-only reverse iterator.
    • RwRevIter: A read-write reverse iterator.

    These iterators support the same iteration methods (move_between_keys, move_through_duplicate_values) and codec remapping as their forward counterparts. They start from the last element of the database and move backwards using next() and last().

  11. Configure TLS usage for transactions

    main

    When opening transactions, you can specify whether to use Thread Local Storage (TLS). This choice affects performance and thread safety.

    • WithTls:
      • Performance: Often faster.
      • Thread Safety: !Send. Transactions cannot be moved between threads.
      • Constraint: A thread can only have one active WithTls transaction at a time.
    • WithoutTls:
      • Performance: Slightly slower.
      • Thread Safety: Send. Transactions can be moved between threads.
      • Constraint: A thread can use any number of WithoutTls transactions simultaneously.

  12. How to choose between TLS and non-TLS read transactions

    main

    The EnvOpenOptions type is generic over T: TlsUsage, which determines how read transactions (RoTxn) behave regarding Thread Local Storage (TLS).

    Using TLS (WithTls)

    This is the default behavior. It is often faster because it uses thread-local storage for reader slots.

    • Constraint: Read transactions are !Send. They cannot be moved between threads.
    • Constraint: A thread can only use one transaction at a time (plus nested transactions). Attempting to use multiple read transactions on the same thread will result in a BadRslot error.
    • Method: Use .read_txn_with_tls() to ensure this mode.

    Without TLS (WithoutTls)

    Use this mode if you need to move transactions between threads.

    • Benefit: Read transactions are Send. They can be moved between threads.
    • Benefit: A single thread can use any number of read transactions simultaneously.
    • Method: Use .read_txn_without_tls() to enable this mode.

    When to use WithoutTls: Use this if your application multiplexes many user threads over individual OS threads, or if you need to pass transactions across thread boundaries.

    // For Sendable transactions (can move between threads)
let mut options = EnvOpenOptions::new().read_txn_without_tls();

// For faster, thread-local transactions (cannot move between threads)
let mut options = EnvOpenOptions::new().read_txn_with_tls();