DOCSEARCHbyAlgolia
Submit Docs

FoundationDB

repository·main·Indexed 12 days ago

https://github.com/apple/foundationdb

A distributed, ordered key-value store providing ACID transactions, designed for high-performance read/write and write-intensive workloads across commodity server clusters. Includes client bindings for Go (supporting API versions 200 through 800) and Java (JDK 8+), as well as monitoring tools like fdbcstat for tracking transaction statistics via eBPF/bcc.

Tokens
304.4K
Snippets
583
Records
1.3K
Agent score
97%

What's inside FoundationDB

  1. Overview of API Tester

    main

    The fdb_c_api_tester is a framework for end-to-end testing of the FoundationDB C API on real clusters. Unlike simulation tests (which are deterministic and single-threaded), API tests are designed to test multi-threaded aspects of the client layers above the Native Client.

    Key Testing Goals:

    • Functional Correctness: Validating the Multi-Version Client (MVC) and Thread-Safe Client.
    • Race Conditions: Detecting issues caused by improper synchronization or accessing Native Client state from wrong threads.
    • Memory Management: Ensuring thread-safe reference counting and verifying that memory allocated by a client library is deallocated by the same library.
    • Interoperability: Ensuring correct behavior across different client API versions.
    • Cluster Upgrades: Verifying that database and transaction state migrate correctly during upgrades and that pending operations are retried successfully.

  2. What is FDB Replay?

    main

    FDB Replay is an interactive Terminal User Interface (TUI) designed for "replaying" FoundationDB simulation trace files (XML format). It is an experimental tool intended to speed up debugging of simulation issues by providing time-based navigation through trace events.

    Instead of manually grepping large XML files, replay allows you to navigate the timeline and see the reflected state of the cluster (topology, DB configuration, recovery state, epoch/version information) at any specific moment in the trace.

  3. Overview of FoundationDB Major Subsystems

    main

    The FoundationDB codebase is partitioned into 12 major subsystems based on data flow and runtime relationships. Understanding these subsystems helps developers navigate the repository and understand how different components (like the Client Library, Transaction Commit Pipeline, and Storage Servers) interact to provide a distributed transactional KV store.

    Key subsystems include:

    • Flow Runtime: The async actor framework and event loop.
    • RPC & Transport: Messaging and peer management.
    • Client Library: The Transaction API used by applications.
    • Transaction Commit Pipeline: Orchestrates version assignment and conflict detection.
    • Storage Server & Engines: Handles reads and applies mutations from the log.
    • Data Distribution: Manages sharding and rebalancing.
    • Simulation & Testing: Provides deterministic simulation (Sim2) and fault injection.

  4. Overview of FoundationDB Operations

    main

    FoundationDB operations cover the lifecycle of an externally accessible cluster, from initial installation and configuration to ongoing administration and disaster recovery. Key operational areas include:

    • Cluster Setup: Installation and configuration of clusters on one or more machines (Note: externally accessible clusters are not supported on macOS).
    • Administration: Managing existing clusters and using the fdbcli command-line tool.
    • Security: Implementing Transport Layer Security (TLS) for authentication and security via public/private key infrastructure.
    • Backup & Recovery: Using the FoundationDB backup tool or disk snapshot-based backup tools for disaster recovery.
    • Monitoring & Status: Interpreting cluster status information via JSON encoding.
    • Advanced Features: Utilizing transaction tagging (for throttling), Testing Storage Server (TSS) for untrusted storage engines, Perpetual Storage Wiggle, and ConsistencyScan.

  5. Overview of FoundationDB capabilities

    main

    FoundationDB is a distributed, multi-model, ordered key-value store designed for large-scale structured data. It provides ACID transactions for all operations and is optimized for both read/write and write-intensive workloads.

    Key characteristics include:

    • Multi-model support: Store various data types within a single database.
    • Scalability and Fault Tolerance: A distributed architecture that scales out and handles hardware failures while maintaining the appearance of a single ACID database.
    • Flexible Architecture: Applications can interact directly with FoundationDB or through stateless 'layers' that add specific capabilities on top of the core engine.

  6. Benchmark TLS performance in peer-to-peer scenarios

    main

    The mtlsbenchmark framework is used to benchmark TLS performance in peer-to-peer network scenarios. It allows you to test different connection patterns using 'Long Running' or 'One Shot' modes.

    Prerequisites

    • OpenSSL or a compatible tool for certificate generation.
    • An environment configured for FoundationDB unit test execution.

  9. CDC Requirements and Limitations

    main

    Capabilities

    • Durable Registrations: Named streams for single key ranges.
    • Consumer API: Once registered, consumers only need the stream name (not the range) to perform reads.
    • Ordered Delivery: Mutation batches are identified by FoundationDB commit versions.
    • Durable Acknowledgements: Controls the retention of TLog history.
    • Proxy Recovery: CDC proxies can be replaced or recover without losing stream ownership or prematurely releasing data.

    Important Limitations

    • No Exactly-Once Guarantees: CDC does not provide exactly-once side effects. Consumers must ensure their output and checkpointed cursors are consistent to achieve exactly-once processing.
    • No Dynamic Range Changes: A stream name is tied to one range. To change the range, you must remove the existing stream and register a new one.
    • No Single-Transaction Acknowledgement: There is no single API to acknowledge a stream and update arbitrary application state in one transaction. This composition must be handled by the consumer.
    • Manual Cleanup Required: Unread CDC-tagged TLog data may grow if a consumer stops. Production environments should implement administrative policies to delete or expire abandoned streams.

  10. FoundationDB Client Development Overview

    main
    FoundationDB provides language bindings for application development against its ordered key-value store. To build applications with FoundationDB, developers should focus on several core areas: installation of local servers for development, understanding general API principles applicable across all bindings, data modeling techniques for key-value representation, and testing client code using workloads. Advanced features include transaction tagging for throttling, transaction profiling for analysis, and automatic idempotency to handle commit_unknown_result errors.

  11. Review FoundationDB feature maturity and status

    main

    FoundationDB features are categorized by their maturity level: production, experimental, or under development.

    Production Features

    • HA/Fearless: Multi-region configuration with satellites for mutation logs. Enables automatic failover to remote regions without data loss (Available since FDB 6.3).
    • DR (Disaster Recovery): Recommended to use HA or three datahall mode instead of traditional DR.
    • RocksDB: Storage engine available in production since FDB 7.3.
    • Redwood: B+ tree implementation with better performance than SQLite (Available since FDB 7.1).
    • Gray failure: Automatic detection and recovery from gray network failures (Available since FDB 7.1).
    • Testing Storage Server (TSS): Allows running a storage server with identical data but different engines to compare performance/correctness (Available since FDB 7.1).
    • Perpetual Storage Wiggle: Gradual replacement of storage servers for space reclamation or engine migration.
    • FDB K8s Operator: Kubernetes operator (Available since FDB 6.3).
    • mTLS: Mutual TLS between clients and FDB servers (Available since FDB 6.1).

    Experimental & Under Development

    • Bulk Loading & Dumping: Dumping idle clusters to S3 and loading from S3 into empty clusters (Available since 7.4).
    • V2 backup (partitioned mutation logs): New backup worker role pulls mutations from tlogs to S3 (Targeting 7.4 for production).
    • Encrypted Backup Files: Use an encryption key to encrypt backup files.
    • Parallel Restore: Restoring V2 backup files (Not fault tolerant for production).
    • Sharded RocksDB Storage Engine: Uses column families for shards to enable file-based data copies.
    • gRPC: Adding gRPC endpoints for client/server communications like fdbcli usage and file transfers.
    • Version Vector: Scaling the transaction system to reduce broadcast to all tlogs.
    • Tag Throttling: Limits transaction rates for specific transaction tags.

  12. Use FDB Metadata Audit Tools for diagnostics and recovery

    main

    The FDB Metadata Audit Tools are used for diagnosing and recovering from FoundationDB metadata corruption involving serverList, keyServers, and serverKeys.

    Warning: These tools are intended for rare metadata inconsistencies and should not be used during normal operations. If you encounter issues on a current release, please file a bug.

    Core Workflow

    1. Check for corruption: Use the check command for read-only diagnostics.
    2. Backup: Always run the backup command to create a JSON snapshot before attempting any repairs.
    3. Repair/Restore: Use restore to roll back after a failed repair, or repair-coalesce to fix uncoalesced KRM entries.

    Prerequisites

    • libfdb_c: The FDB C client library (libfdb_c.so or libfdb_c.dylib).
    • Python 3.6+: The fdb Python bindings are bundled with the tools.

    Finding your Cluster File and libfdb_c

    Run ps auxw | grep fdbserver to find the running process.

    • The Cluster file is the path following the -C argument.
    • The libfdb_c location is found by replacing bin/fdbserver in the path with lib/ (e.g., /usr/local/foundationdb/lib/).

    If libfdb_c is already in your LD_LIBRARY_PATH, you only need the cluster file path.

    # Check for corruption
./metadata-audit.sh check -C /path/to/fdb.cluster

# Backup metadata
./metadata-audit.sh backup -C /path/to/fdb.cluster --output-dir /tmp/meta_backup

# Restore from backup (dry-run)
./metadata-audit.sh restore --backup-dir /tmp/meta_backup_timestamp --dry-run -C /path/to/fdb.cluster