repository·master·Indexed 20 days ago
https://github.com/jnwatson/py-lmdbA universal Python binding for the LMDB (Lightning Memory-Mapped Database) storage engine. It provides high-performance, memory-mapped database access with support for ACID transactions, zero-copy reads via buffers, and duplicate-sort databases. Compatible with CPython >= 3.9 and PyPy, it offers both C extension and CFFI variants. The library supports advanced features such as named databases, fine-grained navigation with Cursors, and writemap mode for increased performance.
py-lmdb is a universal Python binding for the LMDB (Lightning Memory-Mapped Database). It provides a Python interface to the high-performance LMDB storage engine.
For detailed technical documentation, refer to the official LMDB documentation at https://lmdb.readthedocs.io.
LMDB is a high-performance, embedded, key-value store. Use the following comparison to decide if it fits your needs:
| Feature | py-lmdb | pickle | dbm | SQLite | Redis | RocksDB |
|---|---|---|---|---|---|---|
| ACID transactions | Yes | No | No | Yes | No | Yes |
| Concurrent readers | Lock-free | No | No | WAL mode | Yes | Yes |
| Read performance | Excellent | Poor | Fair | Good | Good | Good |
| Write performance | Good | Poor | Fair | Good | Excellent | Excellent |
| Larger than RAM | Yes | No | Yes | Yes | No | Yes |
| Embedded (no server) | Yes | Yes | Yes | Yes | No | Yes |
| Multi-process safe | Yes | No | No | Yes | N/A | Yes |
| Zero-copy reads | Yes | No | No | No | No | No |
When building asynchronous applications, avoid calling LMDB directly from the main event loop. While calling LMDB synchronously might appear safe in specific scenarios (e.g., if the database is entirely in RAM, writes are uncontended, or disk IO is extremely fast), it is a best practice to design your application to handle slow IO.
To maintain responsiveness, offload LMDB operations to a thread pool so that slow IO does not block the main loop's select loop. This protects the application against future performance degradation as the database grows or hardware changes.
By default, each key maps to exactly one value. By opening a database with dupsort=True, you allow a single key to have multiple values, stored in sorted order. This is ideal for one-to-many relationships (e.g., tags for a document).
Duplicate values are stored in a nested B-tree and are sorted lexicographically. Note that the maximum size of a duplicate value is limited to 511 bytes (matching the maximum key size).
env = lmdb.open('/tmp/test', max_dbs=1)
db = env.open_db(b'edges', dupsort=True)
with env.begin(write=True, db=db) as txn:
txn.put(b'node1', b'node2')
txn.put(b'node1', b'node3')
txn.put(b'node1', b'node4')
txn.put(b'node2', b'node5')To avoid copying data between the kernel, the library, and your application, you can request memoryview objects instead of bytes. This is achieved by passing buffers=True to Environment.begin() or Transaction.begin().
Important Safety Rules:
memoryview objects are only valid as long as the producing transaction remains unchanged and uncommitted.txn.put(), txn.delete()) or the end of the transaction invalidates existing buffers.bytes(buf).# Requesting buffers
txn = env.begin(buffers=True)
buf = txn.get(b'somekey')
# Using the buffer
print(len(buf))
print(buf[0])
# Converting to bytes to preserve it
value = bytes(buf)
# Safety example: copying before a write
with env.begin(write=True, buffers=True) as txn:
buf = txn.get(b'foo') # valid until next write
buf_copy = bytes(buf) # valid forever
txn.delete(b'foo') # write invalidates 'buf'
print(buf) # ERROR! invalidated
print(buf_copy) # OKTo scale LMDB performance beyond the GIL, you have two primary patterns:
Multiprocessing (Current Standard): Use multiple processes sharing a single LMDB environment via shared memory mapping. This is the canonical way to scale. It is highly efficient because the page cache is shared, but you must pay the cost of pickling/serializing data across process boundaries.
Free-threading (Incremental Convenience): Use multiple threads within a single process on a free-threaded CPython build. This allows you to scale the Python-level overhead across cores and read large values directly into in-process memory without IPC or serialization overhead. However, this requires explicit support from the C extension to be safe.
Note: Both methods benefit from the fact that py-lmdb already releases the GIL during the actual LMDB C calls (I/O and B-tree work), meaning I/O parallelism is already available in both models.
Proper transaction lifecycle management is critical to prevent two specific issues:
Best Practice: Always wrap Transaction objects in a with statement to ensure they are correctly finalized even if an exception occurs.
# Correct pattern to ensure finalization
with env.begin() as txn:
if txn.get(b'foo'):
# Even if this crashes, txn will be correctly finalized.
do_something()As of the current investigation, py-lmdb does not explicitly declare free-threading support.
If you import py-lmdb on a free-threaded CPython interpreter (e.g., python3.13t or 3.14t with Py_GIL_DISABLED):
RuntimeWarning.PYTHON_GIL=0 or -X gil=0, the module will run without the GIL, but this is unsafe and will lead to data races and crashes because the internal bookkeeping assumes GIL atomicity.Free-threading provides an incremental convenience for read-heavy, multi-threaded, in-process workloads performing many small operations. It allows you to scale the 'Python glue' across cores and avoids the IPC/serialization overhead required when using the traditional multiprocessing approach to scale LMDB.
A Cursor provides fine-grained navigation over key-value pairs. Cursors are created from a transaction and share its lifetime. Use a context manager to ensure the cursor is properly closed.
with env.begin() as txn:
with txn.cursor() as cur:
# use the cursor ...To prevent crashes when objects are invalidated (e.g., via Transaction.abort), py-lmdb ensures child objects do not access memory of deleted resources:
PyObject structures to manage dependencies without extra heap allocation or weakref overhead._deps dictionary mapping dependent object IDs to the corresponding objects. During invalidation, _deps is walked to notify dependents. To prevent invalid native calls, native handles are replaced with a magic object Some_LMDB_Resource_That_Was_Deleted_Or_Closed. Attempting to use this object in a native call will raise a TypeError.LMDB allows you to create multiple named databases within a single environment.
Requirements:
lmdb.open() or initializing lmdb.Environment, you must provide the max_dbs= parameter. This must be done by the first process or thread that opens the environment.Environment.open_db().Environment.dbs().Warning: Environment.dbs() works by attempting to open keys in the main database as named databases. To ensure reliable results, do not mix application keys in the main database with named databases.
The binding is implemented in two ways to balance compatibility and performance: