DOCSEARCHbyAlgolia
Submit Docs

immudb Documentation

repository·master·Indexed 27 days ago

https://github.com/codenotary/immudb

An immutable database with built-in cryptographic proof and verification supporting key-value, document, and relational (SQL) models. Features include a PostgreSQL wire protocol server, structured audit logging, and tamper-evident systems for sensitive data. Documentation covers installation via binary, Docker, and Helm, S3 storage configuration, the immuclient CLI, and mitigation strategies for the 'Linear Fake' vulnerability.

Tokens
59.9K
Snippets
97
Records
419
Agent score
94%

What's inside immudb

  1. Understand immudb proof data structures

    master

    immudb uses a combination of Merkle Trees and linear chains to provide verifiable data integrity. Understanding these structures is essential for verifying that data has not been tampered with.

    Key Data Structures

    • Main Merkle Tree: A persistent tree where each leaf is a transaction hash (Alh). It provides logarithmic complexity for proofs of historical data.
    • Internal Merkle Tree: A transient tree built per transaction from its Key-Value-Metadata entries. Its root hash is EH (Entries Hash).
    • Linear Proof (Accumulated Linear Hash - Alh): A linear chain of transaction hashes. Each transaction's Alh is derived from the previous transaction's Alh, creating a continuous chain of the entire database history.

    Verification Strategy

    To optimize performance, immudb uses the Main Merkle Tree to prove history up to a specific transaction (BlTxID), and then uses the Linear Proof to bridge the gap from that point to the current state. This combines logarithmic complexity for old history with linear complexity for recent, high-velocity writes.

  2. Understand the PostgreSQL compatibility roadmap

    master

    immudb is implementing PostgreSQL compatibility through two main architectural parts:

    1. Part A: pg_catalog as real system tables: Implementing real system tables (like pg_class, pg_attribute, etc.) to allow tools like psql and pgAdmin to function.
    2. Part B: AST-based SQL rewriter: Replacing regex-based query transformations with a robust Abstract Syntax Tree (AST) rewriter that parses PostgreSQL dialect, walks the AST, rewrites constructs, and feeds the result to the immudb parser.

    Compatibility is being rolled out in phases (A1-A5 for catalogs, B1-B2 for the rewriter) and can be enabled via the --pg-catalog-v2 feature flag.

  3. Important SQL Design Constraints

    master

    When developing SQL features for immudb, adhere to these constraints:

    • Immutability: immudb is append-only. Mutations create new versions. Never bypass tx.doUpsert or tx.deleteIndexEntries. DELETE is a soft delete.
    • LIKE Syntax: LIKE and ILIKE use Go regex syntax, not standard SQL %/_ wildcards. For example, use 'hello.*' instead of 'hello%'.
    • Catalog Persistence: Metadata that must survive restarts requires a key prefix in stmt.go (e.g., catalogSequencePrefix = "CTL.SEQUENCE.") and implementation of persist* functions to write to the KV store.
    • PG Wire Protocol Dispatch:
      • sql.DataSource implementations follow the query() path (returns rows).
      • All other statements follow the exec() path (no rows returned).
      • RETURNING clauses make DML statements implement DataSource.

  4. PostgreSQL Compatibility Roadmap Overview

    master

    immudb is implementing a two-part architectural upgrade to improve PostgreSQL wire compatibility. Currently, compatibility relies on a 'regex-triage + canned-response' layer which is prone to errors with complex queries or specific clients (like psql, pgAdmin, or Rails).

    The roadmap consists of:

    1. Part A (System Tables): Replacing hardcoded responses with real pg_catalog system tables that walk immudb's live catalog. This allows the SQL engine to handle JOINs, WHERE clauses, and aggregates against the catalog naturally.
    2. Part B (AST Rewriter): Replacing regex-based string manipulation with an Abstract Syntax Tree (AST) based query rewriter driven by a real PostgreSQL parser.

  5. Implement a pg_catalog or information_schema Resolver

    master

    To add virtual tables (like those in pg_catalog), follow these steps:

    1. Define columns: In pkg/pgsql/pgschema/table_resolvers.go, define the column descriptors using sql.ColDescriptor.
    2. Implement the TableResolver interface: Create a struct with:
      • Table(): Returns the table name.
      • Resolve(ctx, tx, alias): Returns a RowReader containing the metadata/rows.
    3. Register the resolver: Add the new instance to the tableResolvers slice in pkg/pgsql/pgschema/table_resolvers.go.
    4. Add tests: Add tests in pkg/pgsql/pgschema/resolvers_test.go.
    // Step 1: Define columns
var myTableCols = []sql.ColDescriptor{
    {Column: "col1", Type: sql.IntegerType},
    {Column: "col2", Type: sql.VarcharType},
}

// Step 2: Implement the resolver
type myTableResolver struct{}

func (r *myTableResolver) Table() string { return "my_table" }

func (r *myTableResolver) Resolve(ctx context.Context, tx *sql.SQLTx, alias string) (sql.RowReader, error) {
    catalog := tx.Catalog()
    tables := catalog.GetTables()

    var rows [][]sql.ValueExp
    for _, t := range tables {
        rows = append(rows, []sql.ValueExp{
            sql.NewInteger(int64(t.ID())),
            sql.NewVarchar(t.Name()),
        })
    }

    return sql.NewValuesRowReader(tx, nil, myTableCols, true, alias, rows)
}

  6. Integrate immudb into your application

    master

    immudb provides native SDKs for several programming languages to facilitate integration. If a native SDK is not available for your language, you can use immugw to interact with immudb via a REST API.

    Available SDKs

    Getting Started

  7. Optimize integer encoding for signed values

    master

    When defining or using integer fields in immudb protobuf messages, choose the type based on the expected value range to optimize encoding efficiency:

    • Negative Numbers: Standard int32 and int64 use variable-length encoding that is inefficient for negative numbers. Use sint32 or sint64 instead, as they are designed to encode signed values more efficiently.
    • Large Unsigned Values: If uint32 values are often greater than $2^{28}$, use fixed32. If uint64 values are often greater than $2^{56}$, use fixed64. These types always occupy 4 or 8 bytes respectively and can be more efficient for large values.

  8. Enable Amazon S3 storage for immudb

    master

    immudb can use Amazon S3 (or S3-compatible storage like MinIO) as its storage backend. This is configured via environment variables.

    To use AWS IAM roles instead of static access keys, enable IMMUDB_S3_ROLE_ENABLED. If running on AWS Fargate, you can use IMMUDB_S3_USE_FARGATE_CREDENTIALS=true to automatically source credentials.

    export IMMUDB_S3_STORAGE=true
export IMMUDB_S3_ACCESS_KEY_ID=<S3 ACCESS KEY ID>
export IMMUDB_S3_SECRET_KEY=<SECRET KEY>
export IMMUDB_S3_BUCKET_NAME=<BUCKET NAME>
export IMMUDB_S3_LOCATION=<AWS S3 REGION>
export IMMUDB_S3_PATH_PREFIX=testing-001
export IMMUDB_S3_ENDPOINT="https://${IMMUDB_S3_BUCKET_NAME}.s3.${IMMUDB_S3_LOCATION}.amazonaws.com"

./immudb

  9. Optimize S3 upload performance by disabling upload verification

    master
    By default, immudb skips the Exists round trip and uses a lazyRemoteReader to improve upload speed. This is safe for modern S3-compatible backends that provide read-after-write consistency. If you require explicit verification of every upload, you can opt back into the legacy two-round-trip path using WithVerifyUploads(true).
    WithVerifyUploads(true)

  10. Add new SQL syntax and keywords

    master

    To introduce new SQL keywords or statements, follow these steps:

    1. Add keyword to the lexer: In embedded/sql/parser.go, add the keyword to the keywords map.
    2. Declare the token in the grammar: In embedded/sql/sql_grammar.y, declare the new token using %token <keyword> ... MYNEWKW.
    3. Add grammar rules: Define the syntax rules in embedded/sql/sql_grammar.y and assign them to a parent rule like ddlstmt, dmlstmt, or dqlstmt.
    4. Implement the statement type: In embedded/sql/stmt.go, create a struct that implements the SQLStmt interface (including readOnly(), requiredPrivileges(), inferParameters(), and execAt()).
    5. Regenerate the parser: Run the goyacc command to update embedded/sql/sql_parser.go.
    6. Add tests.
    # Step 5: Regenerate the parser
go run golang.org/x/tools/cmd/goyacc -l -o embedded/sql/sql_parser.go embedded/sql/sql_grammar.y

  11. Build immudb Docker images

    master

    To build your own container images for immudb, immuadmin, and immuclient, use the provided Dockerfiles from the root of the repository.

    docker build -t myown/immudb:latest -f Dockerfile .
docker build -t myown/immuadmin:latest -f Dockerfile.immuadmin .
docker build -t myown/immuclient:latest -f Dockerfile.immuclient .

  12. Install and use immuclient

    master

    The immuclient is the command-line interface for interacting with immudb. You can download the binary from GitHub or run it via Docker. Once running, you can enter an interactive shell or run commands directly.

    # Via Binary
wget https://github.com/codenotary/immudb/releases/download/v1.5.0/immuclient-v1.5.0-linux-amd64
mv immuclient-v1.5.0-linux-amd64 immuclient
chmod +x immuclient

# Start interactive shell
./immuclient

# Or use Docker
docker run -it --rm --net host --name immuclient codenotary/immuclient:latest