montydb

repository·master·Indexed 20 days ago

https://github.com/davidlatwe/montydb

A pure Python implementation of a database that mimics the MongoDB API, designed for environments where running a full MongoDB instance is difficult. It supports multiple storage engines including In-Memory, Flat-File, SQLite, and LMDB (Lightning). The library provides a MongoDB-like interface via MontyClient, MontyDatabase, and MontyCollection for CRUD operations, as well as utilities for importing and exporting data in JSON and BSON formats.

Tokens
10.6K
Snippets
46
Records
56
Agent score
69%

What's inside montydb

  1. Install montydb

    master

    You can install montydb using pip or uv.

    To use real bson (which installs pymongo), use the [bson] extra. This is recommended if you need full BSON support beyond the built-in ObjectId implementation.

    To use the LMDB storage engine, use the [lmdb] extra.

    # Standard installation
    pip install montydb
    
    # Using uv
    uv pip install montydb
    
    # With real BSON support (installs pymongo)
    pip install montydb[bson]
    
    # With LMDB support
    pip install montydb[lmdb]
  2. Configure CursorType for tailable or exhaustible cursors

    master

    The CursorType class defines flags used to control how a cursor behaves during iteration, particularly for streaming or specialized data access patterns.

    • CursorType.NON_TAILABLE (0): Standard cursor behavior.
    • CursorType.TAILABLE (2): A cursor that stays open at the end of the result set, waiting for new data.
    • CursorType.TAILABLE_AWAIT (34): A tailable cursor that blocks/awaits new data when the result set is empty.
    • CursorType.EXHAUST (64): A cursor that continues until all data is consumed.
    # Example of setting a specific cursor type if supported by the collection
    cursor = collection.find({"log": "info"}, cursor_type=CursorType.TAILABLE_AWAIT)
  3. Collection name constraints and invalid characters

    master

    When creating or accessing collections, the following rules apply to the name string:

    • Forbidden Characters: Names must not contain $, \0, or \x00.
    • Forbidden Prefixes: Names must not start with system..
    • Blank Names: Names cannot be empty or blank.

    Violating these rules will result in an errors.OperationFailure.

  4. Configure Flat-File storage

    master

    The flatfile engine is the default on-disk storage. You can configure it using set_storage to control the cache behavior.

    FlatFile specific settings:

    • cache_modified: Number of document CRUD operations to cache before flushing to disk.
    from montydb import set_storage, MontyClient
    
    # Configure flatfile storage with a cache size of 5
    set_storage("/db/repo", storage="flatfile", cache_modified=5)
    
    # Initialize client
    client = MontyClient("/db/repo")
  5. Access sub-collections via dot notation

    master

    A MontyCollection can be used to access nested or sub-collections using the __getitem__ syntax. If you attempt to access an attribute starting with an underscore (e.g., collection._name), it will raise an AttributeError suggesting you use the database indexer instead.

    To access a collection named sub_collection within a database, use database['sub_collection'] or collection['sub_collection'] (which returns a collection with the name parent_collection.sub_collection).

    sub_col = collection['sub_collection']
    # This is equivalent to database.get_collection("parent.sub_collection")
  6. Configure SQLite storage

    master

    SQLite is not the default on-disk engine and must be explicitly configured via set_storage before initializing the client.

    Note: SQLite storage files created with montydb <= 1.3.0 are not compatible with montydb >= 2.0.0.

    SQLite Configuration Options:

    • journal_mode: SQLite pragma (e.g., "WAL").
    • check_same_thread: Connection option. Pass False to allow multi-threaded access.
    • synchronous: Write concern (integer).
    • automatic_index: Boolean.
    • busy_timeout: Milliseconds.
    from montydb import set_storage, MontyClient
    
    repo = "/db/repo"
    set_storage(
        repository=repo,
        storage="sqlite",
        use_bson=True,
        journal_mode="WAL",
        check_same_thread=False,
    )
    
    client = MontyClient(
        repo,
        synchronous=1,
        automatic_index=False,
        busy_timeout=5000
    )
  7. Configure LMDB (Lightning) storage

    master

    The lightning engine (LMDB) is not the default and must be configured via set_storage before initializing the client.

    LMDB specific settings:

    • map_size: The maximum size (in bytes) the database may grow to.
    from montydb import set_storage, MontyClient
    
    set_storage("/db/repo", storage="lightning", map_size=10485760)
    client = MontyClient("/db/repo")
  8. Basic CRUD usage with MontyClient

    master

    The MontyClient provides a MongoDB-like interface in pure Python. You can access collections via the .db attribute and perform standard CRUD operations like insert_many and find using MongoDB query operators (e.g., $gt).

    from montydb import MontyClient
    
    # Initialize an in-memory client
    client = MontyClient(":memory:")
    
    # Access a collection
    col = client.db.test
    
    # Insert documents
    col.insert_many([{"stock": "A", "qty": 6}, {"stock": "A", "qty": 2}])
    
    # Query documents using MongoDB operators
    cur = col.find({"stock": "A", "qty": {"$gt": 4}})
    
    # Iterate through results
    print(next(cur))
    # Output: {'_id': ObjectId('...'), 'stock': 'A', 'qty': 6}
  9. Import, Export, and Restore data

    master

    The montydb.utils module provides tools for data migration and backups using JSON or BSON formats.

    • montyimport: Imports content from an Extended JSON file into a MontyCollection.
    • montyexport: Produces a JSON export of a MontyCollection.
    • montyrestore: Loads a binary BSON dump into a MontyCollection.
    • montydump: Creates a binary BSON export from a MontyCollection.
    from montydb import open_repo, utils
    
    # Exporting data
    with open_repo("foo/bar"):
        utils.montyexport("db", "col", "/data/dump.json")
    
    # Importing data
    with open_repo("foo/bar"):
        utils.montyimport("db", "col", "/path/dump.json")
    
    # Binary dump/restore
    with open_repo("foo/bar"):
        utils.montydump("db", "col", "/data/dump.bson")
    
    with open_repo("foo/bar"):
        utils.montyrestore("db", "col", "/path/dump.bson")
  10. Record MongoDB queries with MongoQueryRecorder

    master

    The MongoQueryRecorder allows you to record MongoDB query results over a period of time by accessing the database profiler. It reproduces find and distinct commands.

    Requirements: Requires pymongo and access to the database profiler.

    from pymongo import MongoClient
    from montydb.utils import MongoQueryRecorder
    
    client = MongoClient()
    recorder = MongoQueryRecorder(client["mydb"])
    recorder.start()
    
    # ... run your application or queries ...
    
    recorder.stop()
    results = recorder.extract()
    # results is a dict: {<collection_name>: [<doc_1>, <doc_2>, ...], ...}