repository·main·Indexed 12 days ago
https://github.com/benbjohnson/litestreamA standalone disaster recovery tool for SQLite that provides incremental replication of database changes to S3 and other file destinations. It can be used as a CLI or integrated into Go applications as a library, supporting backends including S3, Google Cloud Storage, Azure Blob Storage, and local filesystems.
Litestream is a standalone disaster recovery tool designed for SQLite. It operates as a background process that safely replicates database changes incrementally to a secondary location, such as another file or an S3-compatible object store.
Because Litestream communicates with SQLite exclusively through the official SQLite API, it is designed to be safe and will not corrupt your database.
The Litestream VFS (Virtual File System) is a SQLite extension that enables applications to interact directly with Litestream replica storage (such as S3, GCS, or Azure Blob) without needing to restore the database to a local disk first.
LITESTREAM_REPLICA_URL.SQLite reserves a special lock page at exactly 1GB (0x40000000 bytes). This page cannot contain data and must be skipped during replication. When testing or implementing logic that interacts with large databases, ensure that replication logic correctly identifies and skips this page.
To test this, you can use the litestream-test tool to populate a database that exceeds this threshold.
# Use litestream-test tool for large databases
./bin/litestream-test populate \
-db test.db \
-target-size 1.5GB \
-page-size 4096SQLite reserves a special page at exactly 1,073,741,824 bytes (0x40000000) for locking. This is known as the Lock Page.
Critical Rules for Developers/Replicators:
page_size.Calculating the Lock Page Number:
LockPageNumber = (0x40000000 / pageSize) + 1
Examples:
262145131073655373276916385const PENDING_BYTE = 0x40000000 // 1GB mark
func LockPageNumber(pageSize int) uint32 {
return uint32(PENDING_BYTE/pageSize) + 1
}An LTX file consists of four main sections arranged sequentially:
The total file size is calculated as:
FileSize = HeaderSize + (PageCount * (PageHeaderSize + PageSize)) + PageIndexSize + TrailerSize.
Checkpointing is the process of moving data from the WAL file back into the main database file. SQLite supports several modes:
PASSIVE: Non-blocking; it fails if there are active readers.FULL: Waits for readers to finish; blocks new readers from starting.TRUNCATE: Similar to FULL, but also truncates the WAL file back to zero size.Note: The RESTART mode has been removed (due to issue #724) because it was found to be write-blocking.
Litestream's Checkpoint Strategy:
TruncatePageN $\rightarrow$ Use TRUNCATE (emergency).MinCheckpointPageN $\rightarrow$ Use PASSIVE.CheckpointInterval has elapsed $\rightarrow$ Use PASSIVE.Litestream is organized into several layers of abstraction to separate CLI concerns, core logic, and storage backends:
store.go: Coordinates multiple databases and schedules compaction.db.go: Manages a single SQLite database, WAL monitoring, and checkpoints.replica.go: Handles replication mechanics to a single destination and tracks position.ReplicaClient interface.To ensure correct system behavior, separate database state management from replication mechanics.
db.go): Responsible for database state, restoration, and monitoring. This layer must handle logic for when the database is behind the replica (e.g., clearing local L0 cache and fetching the latest L0 file).replica.go): Responsible strictly for replication mechanics. It should not contain logic for checking database state or initiating restorations.ReplicaClient implementations.// CORRECT - DB layer handles database state
func (db *DB) init() error {
if db.needsRestore() {
if err := db.restore(); err != nil {
return err
}
}
// Then start replica for replication only
return db.replica.Start()
}
func (r *Replica) Start() error {
// Replica focuses only on replication
return r.startSync()
}When writing unit tests that shouldn't depend on real storage or filesystems, use mock implementations.
Use a MockReplicaClient to simulate network behavior. It allows you to control:
FailureRate: Probability of a simulated error.Latency: Artificial delay for operations.EventualDelay: Delay before a file becomes visible to simulate eventual consistency.Use a MockDB to control database-level behaviors like SyncDelay or CheckpointFailures.
type MockReplicaClient struct {
mu sync.Mutex
files map[string]*ltx.FileInfo
data map[string][]byte
// Control behavior
FailureRate float64
Latency time.Duration
EventualDelay time.Duration
}Cloud storage providers (S3, R2) are eventually consistent. To avoid reading partial or stale data during compaction:
Always attempt to read from the local disk first. If the local file exists, use it as it is guaranteed to be complete and consistent. Only fall back to the remote storage if the local file is missing.
// CORRECT - Check local first during compaction
f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))
if err == nil {
// Use local file - it's complete and consistent
return f, nil
}
// Only fall back to remote if local doesn't exist
return replica.Client.OpenLTXFile(...)The Page Index is a binary search tree that enables efficient random access. Instead of parsing raw bytes manually, use ltx.DecodePageIndex to obtain a map of page numbers to ltx.PageIndexElem objects.
Each PageIndexElem contains:
Level: The tree level.MinTXID / MaxTXID: The transaction ID range.Offset: Byte offset of the encoded payload.Size: Bytes occupied by the payload.