DOCSEARCHbyAlgolia
Submit Docs

bbolt Documentation

repository·main·Indexed 27 days ago

https://github.com/etcd-io/bbolt

bbolt is a high-performance, pure Go embedded key/value store designed for simplicity and reliability. A fork of Ben Johnson's Bolt, it provides a B+tree-based storage engine with fully serializable ACID transactions, supporting databases up to 1TB. The documentation covers core API entry points like Open(), DB.Begin(), and Bucket.Put(), as well as a comprehensive command-line utility for database inspection, integrity checks, statistics, and recovery surgery.

Tokens
11.4K
Snippets
20
Records
111
Agent score
90%

What's inside bbolt

  1. Overview of bbolt

    main
    bbolt is a pure Go key/value store inspired by LMDB. It is designed to be a simple, fast, and reliable embedded database for projects that do not require a full database server (like Postgres or MySQL). It is a fork of Ben Johnson's Bolt, maintained to provide improved reliability, stability, bug fixes, and performance enhancements while preserving backwards compatibility with the Bolt API.

  2. Inspect and manipulate bbolt database files with the CLI

    main

    The bbolt command-line utility allows you to inspect and manipulate bbolt database files.

    Important Note on Locking: A bbolt database file can only be opened by one read-write process at a time because it is exclusively locked when opened. Ensure any processes using the database (such as an etcd instance) are stopped before performing write operations or certain inspections that require exclusive access.

  3. Project status and stability

    main
    bbolt is considered stable with a fixed API and fixed file format. It uses full unit test coverage and randomized black box testing to ensure database consistency and thread safety. It is suitable for high-load production environments and can serve databases as large as 1TB.

  4. Compare bbolt with other database engines

    main

    Relational Databases (Postgres, MySQL)

    • Access Pattern: Bolt uses byte slice keys; relational databases use SQL rows.
    • Architecture: Bolt is an embedded library running in your application process; relational databases are typically standalone servers.
    • Overhead: Bolt avoids network serialization/transport overhead but limits multi-process access.

    LSM-tree Databases (LevelDB, RocksDB)

    • Structure: Bolt uses a B+tree; LevelDB uses an LSM tree.
    • Performance: LevelDB is better for high random write throughput (>10,000 w/sec) or spinning disks. Bolt is better for read-heavy workloads and range scans.
    • Transactions: Bolt supports fully serializable ACID transactions; LevelDB does not (it supports batch writes and snapshots but lacks safe compare-and-swap).

    LMDB

    • Architecture: Both use B+trees and support lock-free MVCC with a single writer and multiple readers.
    • Safety: Bolt prioritizes simplicity and safety, disallowing actions that could corrupt the database (except DB.NoSync). LMDB allows some unsafe direct writes for performance.
    • Memory Management: Bolt handles incremental mmap resizing automatically, whereas LMDB requires a maximum mmap size.

  5. Understand bbolt performance and usage caveats

    main

    Performance Considerations

    • Workloads: Optimized for read-intensive workloads. Sequential writes are fast, but random writes can be slow. Use DB.Batch() or a write-ahead log to mitigate random write latency.
    • Hardware: SSDs provide significant performance boosts over spinning disks due to B+tree random page access.
    • Memory: Bolt uses memory-mapped files. High memory usage is expected as the OS caches the file. It can handle databases larger than physical RAM if the virtual address space allows.
    • Page Utilization:
      • Avoid high Bucket.FillPercent for buckets with random inserts.
      • Use larger buckets to avoid poor page utilization once they exceed the page size (typically 4KB).
      • Bulk loading >100,000 random key/value pairs into a single new bucket in one transaction is not advised.

    Critical Safety & Lifecycle Rules

    • Transaction Scope: Byte slices returned from Bolt are only valid during the transaction. Accessing them after a commit or rollback can cause an unexpected fault address panic.
    • Concurrency: Bolt uses an exclusive write lock on the database file; it cannot be shared by multiple processes.
    • Data Persistence: Deleting large amounts of data does not reclaim disk space immediately; Bolt maintains a free list of unused pages within the file for reuse.
    • Endianness: The data file is endian-specific because data structures are memory-mapped. You cannot move a Bolt file between different endian architectures (e.g., little-endian to big-endian).

  7. Iterate over bucket keys and values using Cursor

    main

    A Cursor is an iterator used to traverse key/value pairs in a Bucket in lexicographical order.

    Important Usage Notes:

    • Transaction Lifecycle: Cursors are obtained from a transaction and are only valid as long as that transaction remains open.
    • Data Validity: The []byte keys and values returned by the cursor are only valid for the life of the transaction.
    • Mutation Warning: Changing data while traversing with a cursor may invalidate it and cause unexpected results. You must reposition your cursor (e.g., by calling First() or Seek()) after mutating data.
    • Nested Buckets: Cursors see nested buckets, but the value returned for a bucket entry will be nil.

  9. Manage database transactions with Tx

    main

    The Tx type represents either a read-only or a read/write transaction.

    • Read-only transactions: Used for retrieving values for keys and creating cursors.
    • Read/write transactions: Used to create/remove buckets and create/remove keys.

    IMPORTANT: You must always call Commit() or Rollback() when finished with a transaction. Long-running read transactions prevent the writer from reclaiming pages, which can cause the database to grow rapidly.

  10. Use bbolt surgery commands for advanced database operations

    main

    The surgery command provides advanced, low-level operations for repairing or modifying bbolt database files. These commands are intended for 'surgery' on the database structure and should be used with caution.

    Note: Most surgery commands require an --output flag to specify a new file path where the modified database will be written, preventing accidental corruption of the source file.

  11. Run synthetic benchmarks with the `bench` command

    main
    The bench command runs synthetic read and write benchmarks against a bbolt database to measure performance. It supports various write and read patterns, profiling, and custom data sizes. If no --path is provided, a temporary file is created and automatically removed after the benchmark completes, unless the --work flag is used.

  12. Troubleshoot known bbolt issues

    main

    Linux ext4 Corruption

    On Linux, bbolt may encounter data corruption if the ext4: fast commit feature (introduced in kernel v5.10) is enabled. Fixes are available in stable LTS patchlevels:

    • 5.10.94+
    • 5.15.17+
    • 5.15.27+
    • 5.17 (non-LTS)

    Zero-length Values

    Writing a value with a length of 0 will always result in reading back an empty []byte{} value.

    Iteration Deletion

    Removing key/value pairs from a bucket while iterating with a Cursor may cause the cursor to skip entries. If you call c.Next() immediately after a removal, it may skip the next key/value pair.