DOCSEARCHbyAlgolia
Submit Docs

lakeFS Documentation

repository·master·Indexed 26 days ago

https://github.com/treeverse/lakefs

lakeFS is an open-source data version control tool providing Git-like capabilities—such as branching, merging, and rollback—for object storage including S3, Azure Blob, and GCS. This documentation covers the lakeFS Hadoop FileSystem for Spark jobs, as well as SDKs for Java, Python (including a high-level wrapper and HTTP API client), and Rust.

Tokens
253.2K
Snippets
515
Records
1.4K
Agent score
90%

What's inside lakeFS

  1. Overview of Delta Table Differ

    master

    The Delta Table Differ is a feature designed to help data engineers debug ETL processes by comparing Delta Lake tables across different lakeFS branches. It allows users to see the history of changes (log entries) applied in a topic branch since it was created from a base branch (a "two-dots" diff, similar to git log branch1..branch2).

    Key Capabilities:

    • Compares metadata changes using the operation history of the two tables.
    • Optionally shows changes in the number of rows.
    • Provides a GUI-based experience within the lakeFS UI.

    Limitations:

    • Diffing is limited to available Delta Log entries (JSON files), which are subject to your Delta Lake data retention settings (default is often 30 days).
    • Log compaction is not supported.
    • Tables must reside at the same path in both branches to be diffable (e.g., repo/branch1/delta/table and repo/branch2/delta/table).

  2. Overview of lakeFS capabilities

    master

    lakeFS is a Data Version Control (Git for Data) tool that transforms object storage (AWS S3, Azure Blob Storage, and Google Cloud Storage) into a Git-like repository.

    Key capabilities include:

    • Isolated Environments: Create branches to get a copy of production data without physical copying, enabling safe ETL testing.
    • Reproducibility: Track data state over time to debug issues, validate ML models, and comply with audits.
    • Write-Audit-Publish (WAP): Implement data quality gates using hooks to ensure only validated data is published to production.
    • Rollback: Use versioning to 'time travel' and revert to a previous state in the event of critical data errors.

  3. Overview of the lakeFS Next Generation Metastore

    master

    The lakeFS Metastore is a proposed solution for managing structured metadata (like tables, partitions, and columns) alongside object-store data. It aims to provide version control for metadata using the same branching and commit capabilities as lakeFS, ensuring that data mutations and metadata mutations can be performed atomically during a merge operation.

    Key Features:

    • Versioned Entities: Metastore entities (databases, tables, etc.) are versioned via lakeFS commits.
    • Atomic Operations: Merges can include both data and metadata changes.
    • Diff-able Metadata: Users can view changes made to the metadata model between commits.
    • Compatibility: Designed to maintain compatibility with Hive Metastore (HMS) Thrift protocols.
    • Future Support: Long-term goals include supporting next-generation table formats like Apache Iceberg and Delta Lake.

  4. Understand the Staging Compaction mechanism

    master

    Staging Compaction is a design proposal for lakeFS to improve read performance for branches with a large number of SealedTokens or many tombstones (deleted entries).

    In the standard model, uncommitted entries are read from the StagingToken and then from SealedTokens in order. As the number of SealedTokens grows, reading a missing entry requires checking every token, which degrades performance. Compaction resolves this by consolidating SealedTokens into a single CompactedMetaRangeID.

  5. Architecture of the Delta Table Differ Plugin System

    master

    The Delta Table Differ is implemented using a Microkernel/Plugin architecture. The lakeFS server acts as the core system, while the Delta diff implementation acts as a plugin.

    Technical Details:

    • Communication Protocol: Uses gRPC via protobufs, allowing plugins to be written in almost any language.
    • Plugin Framework: Based on Hashicorp's go-plugin system.
    • Execution Model: Plugins run as separate subprocesses. This ensures that if a plugin crashes, the core lakeFS server remains stable.
    • Transport: Communication occurs over RPC (supporting both net/rpc and gRPC) via a local network connection.

  6. Use the lakeFS Hadoop FileSystem for Spark jobs

    master
    The lakeFS Hadoop FileSystem is an implementation of org.apache.hadoop.fs.FileSystem. It enables running Spark jobs on lakeFS by performing data operations directly on the underlying storage while using the lakeFS server exclusively for metadata operations. This approach optimizes performance for large-scale data processing.

  7. Transition from Postgres DB to Key/Value (KV) storage

    master

    lakeFS is transitioning its metadata storage from a relational Postgres model to a Key/Value (KV) storage model. This change replaces traditional database tables and locking mechanisms with a KV interface that supports multiple back-end implementations.

    Key Concepts

    • Storage Interface: The kv package provides key/value access to replace current storage and locking handled by Postgres.
    • Data Format: Values are encoded using ProtoBuf or JSON and include version information to facilitate future data migrations.
    • Identity-Based Keys: Keys are derived from the identity and lookup properties of the data.
    • Locking: Database locking is replaced by a set-if (conditional write) mechanism provided by the KV solution.
    • Migration: The implementation includes a migration path from existing Postgres tables to the new KV format within Postgres. This is the first supported migration type.

  8. Understand lakeFS Simplified Authorization (ACLs)

    master

    lakeFS uses an Access Control List (ACL) system to simplify authorization. Instead of managing complex Policy-Based Access Control (PBAC) directly, users interact with Permissions, Scopes, and Grants via a GUI or lakectl.

    Core Concepts

    Permissions

    Permissions define what a group of users can do. lakeFS provides four fixed permissions:

    PermissionAllowsExisting Group Equivalent
    ReadRead operations, creating access keysViewers
    WriteAll data read and write operationsDevelopers
    SuperAll operations except authSuperUsers
    AdminAll operations (global, cannot be scoped)Admins

    Note: All users are automatically granted AuthManageOwnCredentials.

    Scopes

    Permissions (except Admin) can be scoped to specific repositories. An Admin permission is global and cannot be restricted to specific repositories.

    Grants

    An ACL grant consists of assigning a single Permission to a group within a specific Scope (set of repositories).

  9. Understand S3 Client acquisition in Spark clients

    master

    lakeFS uses two primary Spark clients that require access to an S3 client to function correctly. Understanding how these clients acquire S3 access is critical for troubleshooting connectivity and authentication issues in Spark environments.

    Spark Clients and S3 Requirements

    ClientUsageS3 Client Requirement
    Spark Metadata clientGC (committed/uncommitted), Spark Export, direct metadata accessAccesses stored metadata directly on S3 and deletes data objects.
    lakeFSFSReading and writing directly on lakeFSReads ETags of uploaded objects to populate lakeFS metadata.

    Client Acquisition Strategies

    To reduce friction and improve reliability, lakeFS aims to unify how these clients acquire S3 clients using a prioritized list of strategies:

    1. Client 'Stealing' (Preferred): Attempting to extract an existing S3 client from the underlying S3AFileSystem. This is the recommended approach as it leverages the existing Spark/Hadoop configuration (including complex authentication like Databricks credential scopes).
    2. Explicit Creation: Creating a new client using static access keys or STS (Security Token Service).

    Key Implementation Details

    • Configuration: The order of acquisition strategies is configurable via a Hadoop property.
    • Type Handling: Because the underlying S3 client might be a different version or package than expected (especially in managed environments like Databricks), callers should avoid casting the returned client to a specific AmazonS3Client type to prevent ClassCastException. Instead, interactions should be handled via reflection.
    • Caching: To maintain performance, the acquired client is cached by FileSystem so that acquisition logic typically runs only once.

  10. Understand the Parquet-based Garbage Collection (GC) design

    master
    lakeFS uses a Parquet-based approach for Garbage Collection to handle large-scale data version control efficiently. Instead of using a dedicated internal format, lakeFS leverages the Parquet format to perform efficient data retrieval, specifically treating GC as a large-scale anti-join operation. This design aims to optimize the identification of objects that are no longer referenced and can be safely deleted.

  11. Understand Repository States and Error Handling

    master

    lakeFS manages repository lifecycle through specific states to ensure data consistency during creation and deletion. When interacting with the API, you may encounter specific error states based on these transitions:

    • active: The repository is fully created, usable, and all associated entities (branches, commits) are accessible.
    • initial: The repository is in the process of being created. If a repository remains in this state for more than 2 minutes, it is automatically treated as deleted.
    • deleting: The repository is currently undergoing deletion. Operations on a repository in this state will return an error.

    Common Error Codes:

    • ErrRepositoryNotFound: The requested repository does not exist.
    • ErrRepositoryDeleting: The repository exists but is currently in the deleting state.
    • ErrNotUnique: An attempt was made to create a repository with an ID that already exists.

  12. Understand the KV Migration mechanism

    master

    The KV Migration is a transitional process used when upgrading lakeFS from a schema-based PostgreSQL implementation to a key-value (KV) store-based PostgreSQL implementation.

    Key concepts:

    • KV Migration Version: The specific schema version at which the KV migration is triggered. This version succeeds the last standard DB migration version.
    • KV Enabled Flag: A development flag that must be enabled to signal that a KV migration is required. If this flag is disabled, lakeFS behaves using standard DB migration logic.
    • DB Import: The process of creating the initial lakeFS database using a previously exported file, which is used as a component of the KV migration flow.