DOCSEARCHbyAlgolia
Submit Docs

Pebble Documentation

repository·master·Indexed 26 days ago

https://github.com/cockroachdb/pebble

Pebble is a high-performance, LevelDB/RocksDB-inspired key-value store written in Go, serving as the default storage engine for CockroachDB. It features block-based tables, checkpoints, indexed batches, and range deletion tombstones. The library includes specialized components such as arenaskl (a lock-free arena-based Skiplist) and batchskl, as well as an atomic marker system for state durability. It provides tools for managing database format major versions and offers limited forward compatibility with RocksDB 6.2.1.

Tokens
26.9K
Snippets
27
Records
155
Agent score
88%

What's inside Pebble

  2. Overview of batchskl skiplist

    master

    batchskl is a fast, non-concurrent skiplist implementation in Go. It is specifically designed to support forward and backward iteration.

    Important Usage Constraints:

    • Not for general purpose use: The interface is optimized for indexing Pebble batches. Keys and values are stored outside of the skiplist itself, which makes it awkward for standard applications.
    • No Deletion Support: The implementation does not support deletion. To handle deletions, higher-level code must insert deletion tombstones and process them accordingly.

  3. Overview of Range Keys in Pebble

    master

    Pebble supports first-class Range Keys, which map a specific range of the keyspace to a value. This feature is designed to enable efficient, sublinear deletion of contiguous ranges of keys (such as MVCC history) without requiring writes proportional to the size of the range.

    Key capabilities include:

    • Mapping a range of keyspace to a value.
    • Optionally including a suffix that encodes a version (e.g., an MVCC timestamp).
    • Configuring iterators to either surface range keys during iteration or mask point keys at lower MVCC timestamps that are covered by the range keys.

  4. Understand the Pebble Commit Pipeline

    master

    The commit pipeline manages the lifecycle of write batches to ensure high performance and data integrity. It is responsible for writing batches to the Write-Ahead Log (WAL) and applying them to the memtable while maintaining two critical invariants:

    1. Sequential WAL Writes: Batches must be written to the WAL in sequence number order.
    2. Ordered Visibility: Batches must be made visible for reads in sequence number order to ensure atomic visibility (preventing readers from seeing partial batch contents).

    The Four Steps of a Commit

    1. Write to WAL: A fast memory copy of the batch to the WAL.
    2. Apply to Memtable: The most CPU-intensive step where mutations are added to the lock-free skiplist.
    3. Bump Visible Sequence Number: Atomically updating the sequence number that indicates which mutations are visible to readers.
    4. Sync WAL (Optional): The most expensive step in terms of wall-clock time, ensuring durability.

  5. Understand Tombstone Density Compaction Heuristics

    master

    Pebble uses compaction heuristics to manage the buildup of point tombstones, which can degrade read performance (especially range scans). While the current implementation (introduced in cockroachdb#3790) focuses on block-level granularity, the design history includes several approaches for detecting tombstone density:

    1. Tombstone Ratio: A simple threshold (TOMBSTONE_THRESHOLD) where an SSTable is compacted if NumDeletions/NumEntries > threshold. This is often ineffective because it doesn't account for tombstone clusters within large tables or overlaps across multiple tables.
    2. Granular/Sliding Window: Dividing an SSTable into buckets or using a sliding window to detect clusters of tombstones. The current implementation uses block-level granularity to identify these dense areas.
    3. Key Range Statistics: Analyzing tombstone density across a specific key range a->b spanning multiple levels of the LSM tree. This involves using version.Overlaps to identify all SSTables in a range and querying their tombstone/key counts via an Annotator.
    4. Maximum Granularity (Per-Block Stats): Storing per-block tombstone/key counts in the SSTable index block to allow $O(\log n)$ queries for precise tombstone counts within any arbitrary key range.

  6. Understand the Atomic Marker system

    master

    The atomic marker system uses files to encode a string value and a monotonically increasing counter to ensure state durability. By creating a new file for each state change and deleting the old one, the system guarantees that at any point on-disk, exactly one visible marker file represents the current state.

    Filenames follow this pattern: marker.<markerName>.<iteration>.<value>

    • <markerName>: An arbitrary identifier.
    • <iteration>: A zero-padded, 6-digit decimal (e.g., 000001).
    • <value>: The state string being stored.

    Example: marker.foo.000001.alpha

  7. Understand Flushable Ingested SSTables

    master

    Pebble implements a 'Flushable Ingested SSTable' mechanism to prevent forced flushes during SSTable ingestion.

    When ingesting SSTables that overlap with an existing memtable, Pebble avoids blocking the ingestion by 'lazily' adding the SSTs to the LSM as a *flushableEntry in the d.mu.mem.queue. Instead of immediately placing them in the lowest possible level (as a regular ingest would), they are placed in the memtable queue and eventually flushed to their appropriate level (L0-L6).

    This state is maintained in memory and is made crash-resilient via a specialized WAL entry.

  8. Understand Pebble memory management

    master

    Pebble's memory usage is primarily driven by two components:

    1. MemTables: Buffers data that has been written to the Write-Ahead Log (WAL) but not yet flushed to an SSTable.
    2. Block Cache: Provides a cache of uncompressed SSTable data blocks.

    To reduce Go Garbage Collector (GC) pressure and prevent excessively large Go heap sizes (which occur when large Block Caches delay GC triggers), Pebble manages the memory for both MemTables and the Block Cache outside of the standard Go heap using the C memory allocator or mmap via a custom allocator. This allows for more predictable Resident Set Size (RSS) compared to standard Go heap allocation.

  9. Understand Pebble SSTable Format Versioning

    master

    Pebble uses two distinct versioning layers to manage data compatibility:

    1. Format Major Version: Indicates the features supported by the Pebble store (DB) itself. This determines which high-level features (like SetWithDelete or block property collection) are enabled when a store is opened. It acts as a gatekeeper for the maximum allowable SSTable format.
    2. SSTable Table Format: A version written into the footer of each individual SSTable file. This determines how that specific file is interpreted (e.g., LevelDB, RocksDBv2, or Pebble-specific versions).

    Crucial Rule: A Pebble store's FormatMajorVersion implies an upper bound on the TableFormat it can safely handle. You must ensure the binary's FormatMajorVersion is high enough to support the TableFormat of the SSTables being read or written.

  10. Upgrade Pebble database format major versions

    master

    Pebble uses 'format major versions' to manage backwards-incompatible physical file format changes. To opt into new formats, you can set FormatMajorVersion in the Options passed to Open, or upgrade an existing database at runtime using DB.RatchetFormatMajorVersion.

    Note: Format upgrades are permanent.

    If your database does not use a custom comparer, merger, or block property collector, you can also use the pebble CLI tool to upgrade.

    Warning: Only use the CLI tool if no custom comparer/merger/property collector are necessary.

    # WARNING: only use if no custom comparer/merger/property collector are necessary.
go run github.com/cockroachdb/pebble/cmd/pebble@v1.1.3 db upgrade <db-dir>

  11. Install Linux profiling tools (perf and blktrace)

    master

    To profile I/O on Ubuntu AWS instances, install the necessary kernel and tracing tools using apt-get.

    sudo apt-get install linux-tools-common linux-tools-4.4.0-1049-aws linux-cloud-tools-4.4.0-1049-aws
sudo apt-get install blktrace

  12. Reading from Virtual SSTables

    master

    Since virtual SSTables do not exist on disk, reads are redirected to the backing physical SSTable via the table cache.

    Implementation Details:

    • The table cache maps virtual SSTable requests to the appropriate physical Reader using the parent SSTable's file number.
    • The Reader API is updated to support bounds-based reads. For example, NewCompactionIter accepts lower, upper []byte parameters to restrict the iterator to the virtual SSTable's range.
    • Iterators like NewRawRangeKeyIter and NewRawDelIter use lower/upper fields to filter keys during block iteration.