DOCSEARCHbyAlgolia
Submit Docs

node-lmdb

repository·master·Indexed 18 days ago

https://github.com/venemo/node-lmdb

A high-performance Node.js binding for LMDB (Lightning Memory-Mapped Database), a transactional key-value store. Version 0.10.1 provides zero-copy retrieval for strings and binary data, supporting environments (Env), databases (Dbi), transactions (Txn), and cursors for efficient in-process database operations.

Tokens
2.2K
Snippets
9
Records
10
Agent score
13%

What's inside node-lmdb

  1. How the LMDB entities work together

    master

    LMDB is organized into four main entities that follow a specific hierarchy:

    1. Env (Environment): Represents the full database environment. An Env object must be used by only one process, but can be used by multiple threads. You must open() it before use and close() it when finished.
    2. Dbi (Database): A sub-database within an Env. An environment can contain multiple named databases or one unnamed database (using null as the name).
    3. Txn (Transaction): The basic unit of work. Only one write transaction can be open in an environment at a time. env.beginTxn() will block until the previous write transaction is commit()ed or abort()ed. A Txn object must only be accessed by one thread at a time.
    4. Cursor: Used to iterate through multiple keys within a specific Dbi.
    var lmdb = require('node-lmdb');

// 1. Create and open Environment
var env = new lmdb.Env();
env.open({ path: './mydata', mapSize: 2*1024*1024*1024, maxDbs: 3 });

// 2. Open a Database
var dbi = env.openDbi({ name: 'myDb', create: true });

// 3. Use a Transaction
var txn = env.beginTxn();
txn.putString(dbi, 'key', 'value');
txn.commit();

// 4. Cleanup
dbi.close();
env.close();

  2. Handle different data types and encodings

    master

    While node-lmdb provides convenient string, number, and boolean APIs, LMDB internally treats most data as binary sequences.

    Strings

    By default, the string API uses UTF-16. If you need to work with other encodings (like UTF-8), use the getBinary method and convert the resulting Buffer using Node.js's Buffer.toString().

    var buf = txn.getBinary(dbi, key);
var str = buf.toString('utf8');

    Binary (Buffers)

    For raw byte access, use getBinary and putBinary. This is the most flexible way to store complex data or non-UTF-16 strings.

    Complex Objects

    To store JavaScript objects, use JSON.stringify() before put and JSON.parse() after get.

    // Example: Reading UTF-8 from a binary store
var buf = txn.getBinary(dbi, key);
var str = buf.toString('utf8');

  4. Avoid Unsafe Get Methods

    master

    The methods getStringUnsafe(), getBinaryUnsafe(), getCurrentStringUnsafe(), and getCurrentBinaryUnsafe() provide zero-copy access to data.

    WARNING: Data returned by these methods is only valid until the next put operation or the end of the transaction.

    If you are using Node 14+, you must also call env.detachBuffer(buffer) after using the buffer to prevent V8 crashes. Because of these complexities, it is generally recommended to use the standard (copying) methods like getString() or getBinary() unless performance is critical.

  5. Perform asynchronous batched writes

    master

    Use env.batchWrite(operations, [options], callback) to execute multiple write operations in a single asynchronous transaction. This improves performance by delegating the work to a separate thread.

    Operation Formats:

    • Object:
      • db (required): The database to write to.
      • key (required): The key.
      • value (optional): The value to put. If absent, performs a del.
      • ifValue (optional): The value required to match for the operation to proceed.
      • ifExactMatch (optional): If true, ifValue must match byte-for-byte. If false (default), ifValue acts as a prefix.
      • ifKey (optional): The key to use for matching the conditional value.
      • ifDB (optional): The database to use for matching the conditional value.
    • Array:
      • [db, key, value] (Put)
      • [db, key] (Delete)
      • [db, key, value, ifValue] (Conditional Put/Delete)

    Callback Arguments:

    • error: Error object if the transaction failed.
    • results: An array of results corresponding to the input operations:
      • 0: Success.
      • 1: Condition not met.
      • 2: Attempt to delete non-existent key (if ignoreNotFound is enabled).
    env.batchWrite([
    [dbi, key1, Buffer.from("Hello")],
    [dbi, key2, Buffer.from("World")],
    [dbi, key3],
    [dbi, key4, valuePlusOne, oldValue]
], options, (error, results) => {
    if (error) {
        console.error(error);
    } else {
        let didWriteToKey4Succeed = results[3] === 0;
    }
});

  6. Create and configure an Environment

    master

    Use new lmdb.Env() to create an environment. You must call .open(options) before performing any operations.

    Options:

    • path: The directory where the database files will be stored.
    • mapSize: The maximum database size (in bytes).
    • maxDbs: The maximum number of databases allowed in this environment.
    var env = new lmdb.Env();
env.open({
    path: __dirname + "/mydata",
    mapSize: 2*1024*1024*1024,
    maxDbs: 3
});

  7. Perform CRUD operations with Transactions

    master

    All data operations must occur within a transaction (Txn). Create a transaction using env.beginTxn().

    Crucial: You must always call txn.commit() or txn.abort() when finished to release the transaction.

    Retrieval Methods:

    • getString(dbi, key)
    • getBinary(dbi, key)
    • getNumber(dbi, key)
    • getBoolean(dbi, key)

    Storage Methods:

    • putString(dbi, key, value)
    • putBinary(dbi, key, value)
    • putNumber(dbi, key, value)
    • putBoolean(dbi, key, value)

    Deletion:

    • del(dbi, key)
    var txn = env.beginTxn();
var value = txn.getString(dbi, 1);

if (value === null) {
    txn.putString(dbi, 1, "Hello world!");
} else {
    txn.del(dbi, 1);
}

txn.commit();

  8. Open a Database (Dbi)

    master

    Use env.openDbi(options) to open a specific database within an environment.

    Options:

    • name: The name of the database (string). Use null for an unnamed database.
    • create: If true, the database will be created if it does not exist.
    • keyIsBuffer: If true, you can work with Node.js Buffer instances as keys.
    • keyIsUint32: If true, uses an optimization for unsigned 32-bit integer keys. Note: If a database is created with this option, it must always be accessed with this option set.
    var dbi = env.openDbi({
    name: "myPrettyDatabase",
    create: true
});

  9. Iterate through a database using a Cursor

    master

    A Cursor allows you to traverse keys in a database.

    Usage:

    1. Create a cursor: new lmdb.Cursor(txn, dbi, [options]).
    2. Use goToFirst() to start.
    3. Use goToNext() to move through the database.
    4. Check against null to detect the end of the iteration (since keys can be falsy).

    Note on Key Types: You can override the key type returned by the cursor by passing options (like { keyIsBuffer: true }) to the Cursor constructor.

    var cursor = new lmdb.Cursor(txn, dbi);

for (var found = cursor.goToFirst(); found !== null; found = cursor.goToNext()) {
    // 'found' contains the key
    // Use cursor.getCurrentString() or cursor.getCurrentBinary() to get data
}

  10. Import node-lmdb bindings

    master

    The node-lmdb package exports the compiled LMDB bindings directly via module.exports. You can require the package to access the LMDB environment, database, and transaction APIs provided by the underlying C++ implementation.

    const lmdb = require('node-lmdb');

// The exported object contains the LMDB bindings
// e.g., lmdb.open, lmdb.Environment, etc.