Project Nessie Documentation

Nessie GC is a tool designed to clean up orphaned files in a Nessie repository. Running it periodically helps maintain repository cleanliness and prevents unnecessary storage costs.

Key Requirements:

• Java: Requires Java 17 or later.
• Nessie Server: A running Nessie server must be reachable.
• Database: Requires a JDBC-compliant database (PostgreSQL, MariaDB, MySQL, or H2) to store live content sets and deferred deletes. It is recommended to use a persistent database for production.
• Credentials: Requires table-format and object-store credentials for the destructive cleanup phases. These credentials should be scoped to the specific warehouse or prefixes intended for cleanup.

The Nessie Content Generator is a utility designed to populate a Nessie instance with (nonsense) data for testing purposes, such as verifying UI functionality. It can create commits across multiple branches, generate tags, and simulate data over a specific time duration by introducing sleeps between commits.

Key capabilities include:

• Generating a specified number of commits.
• Committing to multiple branches.
• Generating tags based on a probability setting.
• Spreading commit generation over a specified time duration.
• Generating content for multiple tables of various types (Iceberg, Delta Lake, etc.).
• Reading references, commits, and content objects.
• Deleting content objects.
• Creating missing namespaces for content keys at branch HEADs.

Nessie is a cloud-native service built on Java and Quarkus, designed for high scalability and resilience in Docker and FaaS environments.

Components

• Nessie Service: Exposes a set of REST APIs and a browser-based UI.
• Libraries: Integrate Nessie's version control capabilities into common data management technologies (like Spark).

Storage Backends

Nessie features a pluggable storage backend. Supported backends include:

• Amazon DynamoDB
• Google BigTable
• Apache Cassandra
• PostgreSQL (including Cockroach)
• MariaDB
• MySQL
• Local storage using RocksDB

Project Nessie is a Git-inspired data version control system designed for data lakes. It provides cross-table transactions and visibility, allowing you to manage data states similarly to how Git manages code.

Key features include:

• Git-inspired version control: Manage data branches, merges, and commits.
• Cross-table transactions: Ensure atomic updates across multiple tables.
• Open data lake support: Integrates with ecosystem tools like Apache Hive, Spark, Dremio, and Trino.
• Apache Iceberg compatibility: Works natively with Apache Iceberg tables.
• Flexible deployment: Can be run as a Docker image or on Kubernetes.

The Nessie Events SPI (Service Provider Interface) module provides reference implementations for notifying external systems about Nessie events. These implementations are built using Quarkus and can be used as templates for building production-grade subscribers.

Available implementations include:

1. PrintingEventSubscriber: A simple implementation that prints events to the console (for demonstration only; do not use in production).
2. Messaging Subscribers (intended as starting points for real-world implementations):
   • KafkaAvroEventSubscriber: Publishes events to Kafka using Avro serialization and a Schema Registry.
   • KafkaJsonEventSubscriber: Publishes events to Kafka using JSON serialization.
   • NatsJsonEventSubscriber: Publishes events to a NATS stream using JSON serialization.

The Nessie Events Notification System is a feature designed to allow external actors to register as subscribers for events published by Nessie. This enables reactive workflows based on changes within Nessie, such as automatic data cleanup or optimization (e.g., triggering a job when a table is deleted).

Key characteristics of the system:

• Integrated: Deployed as part of Nessie, not as a separate process.
• Simple: No requirements on external systems and no changes to the existing Nessie REST API.
• Agnostic: Designed to be technology-agnostic.
• Extensible: Provides a Service Provider Interface (SPI) for users to implement to receive events.

The nessie-events-quarkus module provides a Quarkus-specific implementation of the Nessie events notification system. It is designed to be independent of other Quarkus modules to improve isolation and facilitate testing.

Key features include:

• Quarkus-specific implementations of nessie-events-service classes.
• Asynchronous delivery mechanisms powered by Vert.x, supporting both non-blocking and blocking modes.
• Optional support for logging, tracing, and metrics during event delivery.

This repository contains the infrastructure and plumbing required to run Gatling simulations against Nessie. It facilitates performance testing by providing Nessie-specific logic, such as injecting a NessieClient into tested actions (e.g., generating a Nessie commit). It also includes infrastructure to optionally push metrics to a Prometheus Push Gateway.

For specific instructions on how to run the actual performance tests, refer to the Simulations README.

The Nessie documentation site is managed using mkdocs. The following directory structure defines how content is organized:

• docs/: Contains non-version-dependent documentation.
• in-dev/: Contains version-dependent documentation for the current "in-development/nightly/snapshot" version. This content eventually becomes the next release.
• bin/: Contains shell scripts for developing, building, deploying, and releasing the site.
• overrides/: Contains mkdocs overrides.
• mkdocs.yml: Defines the site structure and navigation.
• docs/docs/: (Note: Versioned docs are maintained in a separate branch; do not edit these directly in the main branch).

Ignored directories: .cache/, build/, and venv/ are ignored by Git.

In Nessie, the smallest atomic unit of change is a single commit. While relational databases (RDBMS) use a central coordinator to manage transactions, Nessie's transactions are distributed.

A single Nessie commit can represent:

• The changes from a single worker in a distributed system.
• A small or massive amount of data changes.
• An atomic merge operation that groups many commits together (acting as a transaction over multiple transactions).

Unlike Git, Nessie-merges work by replaying the changes from a source branch on top of the target branch, ensuring each commit in the resulting chain has exactly one parent.