BadgerDB

BadgerDB is an embeddable, persistent, and fast key-value (KV) database written in pure Go. It is designed to be a performant alternative to non-Go-based stores like RocksDB, specifically optimized for SSDs.

Key Characteristics:

• Pure Go: No Cgo dependency.
• ACID Transactions: Supports concurrent ACID transactions with Serializable Snapshot Isolation (SSI) guarantees.
• WiscKey Design: Based on the WiscKey paper, it separates keys from values to reduce write amplification and optimize for SSDs.
• Advanced Access: Supports TTL (Time-To-Live) and 3D access (key-value-version) via its Iterator API.
• Scalability: Capable of serving data sets spanning hundreds of terabytes.

The skl (skiplist) implementation in Badger is designed for high performance. Benchmarks indicate that skl outperforms both the older skiplist and slist implementations, as well as a standard Go map protected by a read-write lock, particularly as the fraction of read/write operations shifts towards reads.

# Example benchmark results showing skl performance vs Map
BenchmarkReadWrite/frac_10-8           100000000       26.3 ns/op  # skl
BenchmarkReadWriteMap/frac_10-8        10000000        212 ns/op   # map with RW lock

Badger uses transactions to ensure data consistency. The recommended way to use them is via closure wrappers which handle cleanup automatically.

Read-only transactions (DB.View)

Use db.View() for read operations. You cannot perform writes or deletes inside this closure.

Read-write transactions (DB.Update)

Use db.Update() for operations that modify data. If the closure returns an error, the transaction is discarded. If it returns nil, it is committed.

Common Errors:

• ErrConflict: Occurs during a conflict; you may want to retry the operation.
• ErrTxnTooBig: Occurs when pending writes/deletes exceed limits. In this case, commit the current transaction and start a new one.

// Read-only
db.View(func(txn *badger.Txn) error {
  // read operations
  return nil
})

// Read-write
db.Update(func(txn *badger.Txn) error {
  // write operations
  return nil
})

Badger supports ordered merge operations via MergeFunc. You define a function that takes an originalValue and a newValue and returns the merged result.

1. Define a MergeFunc: func(original, new []byte) []byte.
2. Get a merge operator: db.GetMergeOperator(key, mergeFunc, duration).
3. Add values: m.Add(value).
4. Retrieve the result: m.Get().

The duration parameter specifies how often the merge function is run on values added via the operator.

// Example: Append merge function
func add(originalValue, newValue []byte) []byte {
  return append(originalValue, newValue...)
}

key := []byte("merge")
m := db.GetMergeOperator(key, add, 200*time.Millisecond)
defer m.Stop()

m.Add([]byte("A"))
m.Add([]byte("B"))

res, _ := m.Get() // res is "AB"

Badger uses a two-tier encryption system to balance security and performance:

1. Master Key: A user-provided AES key that encrypts the Data Keys. The length of this key determines the AES variant:

   • 16 bytes: AES-128
   • 24 bytes: AES-192
   • 32 bytes: AES-256 Warning: Always use a cryptographically secure random key; never use predictable strings.

2. Data Keys: Auto-generated keys that encrypt the actual data on disk. These are stored alongside the encrypted data.

Benefits of this model:

• Fast Master Key Rotation: Rotating the master key only requires re-encrypting the small Data Keys, not the entire dataset.
• Automatic Data Key Rotation: Data keys rotate independently of the master key.
• Minimal Performance Impact: The separation allows for efficient key management without massive re-encryption overhead.

Badger supports '3D access' (key-value-version). You can interact with different versions of data in two ways:

1. Iterator API: Provides direct access to specific value versions.
2. Options Configuration: You can specify how many versions to keep per key using the Options struct during setup.

Badger is a pure Go key-value database designed for high performance on modern storage devices (SSDs) and large datasets spanning terabytes.

Its core architecture is based on the WiscKey paper, which utilizes a design that separates keys from values. This separation significantly reduces write amplification compared to traditional LSM trees by storing values in a separate value log, making it highly optimized for SSD-conscious storage.