fjall

repository·main·Indexed 25 days ago

https://github.com/fjall-rs/fjall

A log-structured, embeddable key-value storage engine written in Rust. It provides a thread-safe, LSM-tree-based API supporting range searches, multiple keyspaces, and optional transactional semantics via Single Writer or Optimistic concurrency control. Version 3.1.8.

Tokens
10K
Snippets
15
Records
85
Agent score
81%

What's inside fjall

  1. Use a secondary keyspace as a secondary index for range searches

    main
    You can implement secondary indexing in fjall by using a separate keyspace to store mappings from a non-unique attribute to its primary keys. This pattern allows you to perform range searches over attributes that are not part of the primary key. In this configuration, the secondary keyspace acts as an index that facilitates lookups for values within a specific range.
  2. Configure durability in Fjall

    main

    Fjall is agnostic about durability requirements. By default, operations flush to OS buffers but not to disk (similar to RocksDB).

    To ensure data is definitely durable, call Database::persist with a PersistMode. When the database is dropped, it automatically attempts to persist the journal to disk synchronously using PersistMode::SyncAll.

  3. How transactional modes work in Fjall

    main

    While the standard LSM-tree backing store supports MVCC and repeatable snapshot reads, it cannot perform safe read-modify-write operations without the risk of lost updates. If you require serializable transaction semantics, you must use one of the transactional database implementations:

    1. Single Writer (SingleWriterTxDatabase): Opens a transactional database where only a single write transaction can run at a time. This ensures serializability by literally serializing write transactions.
    2. Optimistic (OptimisticTxDatabase): Opens a transactional database for multi-writer, serializable transactions. It uses optimistic concurrency control, meaning transactions may conflict and must be rerun.
  4. Use a secondary keyspace as a unique index with transactions

    main

    To guarantee that a specific attribute is unique across your dataset, you can implement a unique index pattern using a secondary keyspace and transactions.

    In this pattern:

    1. A secondary keyspace is used to store the mapping between the unique attribute value and the primary key of the main record.
    2. Transactions are used to atomically ensure that the entry in the secondary keyspace is created at the same time as the main record, preventing duplicate attribute values.
  5. Manage memory usage in Fjall

    main

    Memory for loaded data and indexes is managed on a per-block basis and is capped by the block cache capacity.

    Important Considerations:

    • Slices and Lifetimes: When you hold a Slice, it keeps the underlying backing buffer (which may be a block) alive. If you need to keep a value for a long time, copy it into a new Vec<u8>, Box<[u8]>, Arc<[u8]>, or a new Slice using Slice::new.
    • Block Cache Recommendation: It is recommended to configure the block cache capacity to approximately 20-25% of available memory, or more if the entire dataset fits in memory.
    • Write Buffers: Each Keyspace has its own write buffer (Memtable), which is the unit of data flushed into the index structure.
  6. Basic usage of Fjall

    main

    Fjall provides a thread-safe BTreeMap-like API for key-value storage. A Database can contain multiple Keyspaces, where each keyspace is an isolated physical LSM-tree.

    Key operations include:

    • insert(key, value): Write data.
    • get(key): Retrieve data.
    • remove(key): Delete data.
    • prefix(prefix): Iterate over keys with a specific prefix.
    • range(range): Iterate over a range of keys.
    • persist(mode): Manually sync the journal to disk.

    Note: Keys are stored in lexicographic order. For integer keys (like timestamps), use big-endian encoding to ensure predictable ordering.

    use fjall::{Database, KeyspaceCreateOptions, PersistMode};
    
    fn main() -> fjall::Result<()> {
        // A database may contain multiple keyspaces
        // You should probably only use a single database for your application
        let db = Database::builder(path).open()?;
        // TxDatabase::builder for transactional semantics
    
        // Each keyspace is its own physical LSM-tree, and thus isolated from other keyspaces
        let items = db.keyspace("my_items", KeyspaceCreateOptions::default)?;
    
        // Write some data
        items.insert("a", "hello")?;
    
        // And retrieve it
        let bytes = items.get("a")?;
    
        // Or remove it again
        items.remove("a")?;
    
        // Search by prefix
        for kv in items.prefix("prefix") {
            // ...
        }
    
        // Search by range
        for kv in items.range("a"..="z") {
            // ...
        }
    
        // Iterators implement DoubleEndedIterator, so you can search backwards, too!
        for kv in items.prefix("prefix").rev() {
            // ...
        }
    
        // Sync the journal to disk to make sure data is definitely durable
        // When the database is dropped, it will try to persist with `PersistMode::SyncAll` automatically
        db.persist(PersistMode::SyncAll)
    }
  7. Use fjall inside a Tokio runtime with spawn_blocking

    main
    Because fjall is a synchronous library, you should not call its blocking operations directly within an asynchronous task in a Tokio runtime. Instead, use tokio::task::spawn_blocking to offload fjall operations to a dedicated thread pool designed for blocking tasks. This prevents blocking the async executor's worker threads and ensures the application remains responsive.
  8. Use axum-kv to build a JSON REST API for Fjall

    main
    The axum-kv example demonstrates how to integrate fjall with the axum web framework and serde_json to expose a key-value store via a JSON REST API. This pattern allows you to wrap a Fjall instance in an HTTP server to provide remote access to your data.
  9. Perform lazy migrations on JSON objects using compaction filters

    main
    This example demonstrates how to implement lazy migrations for JSON objects within fjall. Instead of migrating all data at once, migrations are applied lazily during the compaction process using compaction filters. This approach allows for background updates of data formats without requiring a massive, blocking migration step.
  10. Use WriteTransaction for single-writer transactions

    main

    A WriteTransaction provides a way to perform serialized, cross-keyspace transactions in fjall. Transactions ensure a consistent view of the database; old data is preserved until no active transactions reference it.

    Important: To prevent performance degradation and resource exhaustion, keep transactions short-lived and avoid holding them indefinitely.

    To commit changes, you must explicitly call .commit(). If the transaction is dropped without calling .commit(), all changes will be rolled back.

    let mut tx = db.write_tx();
    tx.insert(&tree, "key", "value");
    tx.commit()?;