lmdb-js

When overlappingSync: true is enabled (default on non-Windows), LMDB allows disk flushing to occur in parallel with future transactions. This improves performance by not holding the writer lock during the slow flush process.

Managing Durability: There is a distinction between a transaction being committed (visible to other threads) and being flushed (durable on disk).

1. Committed: The promise returned by put or remove resolves when the transaction is committed and visible.
2. Flushed: The db.flushed property (or the .flushed property on a promise if separateFlushed: true is set) resolves when the OS reports the data is truly durable on disk.

Note: overlappingSync is generally not recommended on Windows due to poor performance characteristics with large databases.

let db = open('my-db', { overlappingSync: true });

let written = db.put(key, value);
await written; // Wait for it to be committed (visible)

// Access the value immediately after commit
let v = db.get(key);

await db.flushed; // Wait for the last commit to be fully flushed to disk (durable)

lmdb-js supports an optimized LRFU (Least Recently/Frequently Used) and weak-referencing caching mechanism. Enabling caching can improve performance for large objects with high deserialization costs and provides immediate synchronous access to data after a put operation.

Key Benefits:

• Performance: Faster get operations for frequently accessed or large objects.
• Object Identity: As long as an object is in memory, get will return the exact same object instance (identity correlation).
• Immediate Access: You can get a value immediately after calling put without awaiting the put promise.

Requirements:

• Node.js 14.10+ (or Node v13.0 with --harmony-weak-ref).
• Caching does not apply to getRange queries.

Multi-worker/Process Note: Since the cache is stored separately for each process, use the validated: true flag to ensure in-memory objects match the stored data across workers.

// Enable caching with validation for multi-worker environments
let db = open({
	cache: {
		validated: true,
	},
});

// Immediate access pattern
db.put('hi', 'there');
db.get('hi'); // returns 'there' immediately without awaiting the put

To handle high concurrency across multiple processes, use versioning to perform conditional updates. This ensures a data update only occurs if the entry's current version matches your expected version, preventing race conditions.

1. Enable versioning: Set useVersions: true when calling open().
2. Set a version: Use the version argument in put(key, value, version).
3. Conditional update: Use the ifVersion argument in put(key, value, version, ifVersion) or use the ifVersion(key, expectedVersion, callback) method to wrap multiple operations in a conditional block.
4. Retrieve version: Use getLastVersion() to get the current version of an entry.

// 1. Enable versioning
let myDB = open('my-db', { useVersions: true });

// 2. Conditional write: new version 4, only if previous version was 3
myDB.put('key1', 'value1', 4, 3);

// 3. Conditional block for multiple operations
myDB.ifVersion('key1', 4, () => {
	myDB.put('key1', 'value2', 5); // equivalent to myDB.put('key1', 'value2', 5, 4);
	myDB.put('anotherKey', 'value', 3);
	myDB2.put('keyInOtherDb', 'value');
});

Core Architecture

lmdb-js is designed for synchronous reads and asynchronous writes.

• Synchronous Reads: Because LMDB is a memory-mapped database, reading within a transaction typically does not use I/O (except for potential page faults). This allows for instant synchronous access to values.
• Asynchronous Writes: Committing transactions involves I/O. To maximize throughput, lmdb-js queues asynchronous off-thread write operations. Transactions return a Promise that resolves once data is written and flushed to disk.

Key Features

• Data Translation: Uses optimized native C++ code to translate JS values (primitives, arrays, objects) to/from binary data.
• Crash-Proof Design: Uses default syncing configurations to ensure data integrity even during power loss.
• Optimistic Locking: Supports conditional writes, allowing for atomic operations that depend on previously read data. This enables scaling across multiple processes or threads.
• Compression: Offers optional off-main-thread LZ4 compression, which performs compression on the same thread used for asynchronous writes to minimize main-thread impact.
• Automatic Growth: Automatically manages database file expansion using heuristics to minimize fragmentation.