DOCSEARCHbyAlgolia
Submit Docs

ScalarDB Documentation

repository·master·Indexed 20 days ago

https://github.com/scalar-labs/scalardb

A universal HTAP engine that enables ACID transactions and real-time analytics across multiple diverse databases. It features a Consensus Commit protocol for transactional consistency and a data loader CLI for importing and exporting data in CSV, JSON, and JSONL formats. Available for Java projects via Maven Central (e.g., version 3.19.0).

Tokens
8.2K
Snippets
17
Records
28
Agent score
68%

What's inside ScalarDB

  1. What is ScalarDB?

    master
    ScalarDB is a universal HTAP (Hybrid Transactional/Analytical Processing) engine. It is designed to provide ACID transactions and real-time analytics across diverse databases, reducing the complexity of managing multiple database systems simultaneously.

  2. Overview of the Consensus Commit protocol

    master

    The Consensus Commit protocol is used to ensure transactional consistency. Using this protocol requires a schema change to maintain metadata (txId, old data, record state) and a dedicated coordinator table.

    Protocol Workflow

    1. Initialization: A write set $W$ and a coordinator $C$ are initialized.
    2. Prepare Phase:
      • Create a new transaction ID (UUID).
      • Mark every write in $W$ with the txId and set state to prepared.
      • Perform a Compare-And-Swap (CAS) on every prepared write to store the old value.
      • If all CAS operations succeed, the transaction is prepared. If any fail, abort and rollback.
    3. Commit Phase:
      • CAS the transaction ID to the coordinator table.
      • If this succeeds, the transaction is committed (and the coordinator is committed). If it fails, rollback.
    4. Finalization: Update all writes in $W$ to committed. This step can be performed lazily.

    Failure Handling

    • Client failure before Prepare (Step 3): Clear the transaction from memory.
    • Client failure after Prepare (Step 3) but before Commit (Step 5): The coordinator has no status. A subsequent transaction will detect prepared records without a coordinator status, trigger an abort, and roll back the previous transaction.
    • Client failure after Commit (Step 5): The next transaction will perform a 'rollforward' by committing the records on behalf of the failed transaction.

  3. Quickstart guide for ScalarDB

    master

    To begin using ScalarDB, follow the path relevant to your deployment model:

    1. ScalarDB Core: For basic application setup using Java.
    2. ScalarDB Cluster: For enterprise customers requiring cluster-based functionality via the Java API.
    3. Sample Applications: To see running instances of various sample applications, refer to the official ScalarDB sample list.
    4. Configuration: Consult the specific configuration guides for either ScalarDB Core or ScalarDB Cluster to tune your environment.

  4. Install ScalarDB via Gradle or Maven

    master

    ScalarDB is available on Maven Central. You can add it to your Java project using either Gradle or Maven dependency management.

    Gradle Add the following to your dependencies block:

    Maven Add the following to your pom.xml:

    Note: Ensure you use the version compatible with your environment (e.g., 3.19.0).

    dependencies {
    implementation 'com.scalar-labs:scalardb:3.19.0'
}
    <dependency>
  <groupId>com.scalar-labs</groupId>
  <artifactId>scalardb</artifactId>
  <version>3.19.0</version>
</dependency>

  5. How to run the TLC Model Checker for Consensus Commit

    master

    To verify the Consensus Commit protocol using the TLC Model Checker, follow these steps for either the protocol specification (CCSpec) or the implementation (CC).

    Running CCSpec (Protocol Specification)

    1. Open the spec file (CCSpec.tla) and select TLC Model Checker -> New Model.
    2. Behavior Spec: Select Temporal formula and enter Spec.
    3. Model: Enter R <- r1, r2, r3. Select Set of model values and Symmetry set (this treats $r1, r2, r3$ as interchangeable to speed up checking).
    4. Deadlock: Ensure Deadlock is not selected.
    5. Invariants: Enter TypeOK.
    6. Properties: Enter Spec.
    7. Execute: Click the green arrow to run. Monitor States Found and Distinct States to verify progress.

    Running CC (Implementation Specification)

    1. Open the spec file (CC.tla) and select TLC Model Checker -> New Model.
    2. Behavior Spec: Select Temporal formula and enter Spec.
    3. Model: Enter R <- r1, r2, r3. Select Set of model values and Symmetry set.
    4. Deadlock: Ensure Deadlock is not selected.
    5. Invariants: Enter TypeOK, CCS!TypeOK, and CCS!Consistent.
    6. Properties: Enter Spec and CCS!Spec.
    7. Execute: Click the green arrow to run.

  6. Understand ScalarDB Import Modes (Storage vs. Transaction)

    master

    The behavior of the import command changes based on the ScalarDbMode selected:

    • STORAGE Mode: Maps to TransactionMode.SINGLE_CRUD. This is typically used for high-throughput, non-transactional writes directly to the storage layer.
    • TRANSACTION Mode: Maps to TransactionMode.CONSENSUS_COMMIT. This mode uses ScalarDB's distributed transaction capabilities to ensure atomicity and consistency. When using this mode, the CLI validates that the provided TransactionFactory configuration supports starting and aborting transactions.

  7. Reference: `export` command options and flags

    master

    The following options are available for the export command. Note that several options are deprecated in favor of newer versions; do not use both an old option and its replacement in the same command.

    Core Configuration

    • configFilePath: Path to the ScalarDB properties configuration file. If not provided, it defaults to DEFAULT_CONFIG_FILE_NAME in the current directory.
    • namespace: The ScalarDB namespace of the table.
    • table: The name of the table to export.
    • outputFormat: The desired file format. Supported values: CSV, JSON, JSONL.
    • outputDirectory: The directory where the exported file will be saved. If not specified, it defaults to the current working directory.
    • outputFileName: The name of the exported file. If not specified, a filename is generated using the pattern export.{namespace}.{table}.{timestamp}.{format}.

    Scan and Filter Options

    • partitionKeyValue: A list of column-key-value pairs to define a specific partition key for the export.
    • scanStartKeyValue: The starting key for the scan.
    • scanEndKeyValue: The ending key for the scan.
    • scanStartInclusive: Whether the scan should include the scanStartKeyValue.
    • scanEndInclusive: Whether the scan should include the scanEndKeyValue.
    • projectionColumns: A list of specific columns to include in the export.
    • limit: The maximum number of rows to export.

    Formatting and Performance

    • dataChunkSize: The size of data chunks used during export (must be a positive value).
    • maxThreads: The maximum number of threads to use for the export. Defaults to the number of available processors.
    • delimiter: The delimiter used for CSV format.
    • excludeHeaderRow: Whether to exclude the header row in CSV exports.
    • prettyPrintJson: Whether to use pretty-printing for JSON exports.
    • sortOrders: The sort orders to apply to the exported data.
    # Example of how options might be structured in a CLI call
export --namespace my_ns --table my_table --output-format CSV --config-file-path /path/to/scalar.properties --output-directory ./exports

  9. Deprecated export options and migration guidance

    master

    Several options in the export command are deprecated as of release 3.17.0 and will be removed in release 4.0.0. Use the following replacements:

    Deprecated OptionReplacement / Guidance
    --threadsUse --max-threads instead.
    --start-exclusiveUse --start-inclusive (note the inverted logic).
    --end-exclusiveUse --end-inclusive (note the inverted logic).
    --include-metadata, -mUse the configuration property scalar.db.consensus_commit.include_metadata.enabled to control transaction metadata inclusion.

  10. Reference: Cosmos DB schema operation modes

    master

    The schema loader supports three primary modes for managing tables. These are mutually exclusive within the command execution logic.

    --delete-all, -D  Delete tables
--repair-all       Repair tables: repairs the table metadata of existing tables and repairs stored procedure attached to each table
--alter, -A        Alter tables: adds new columns and creates/deletes secondary indexes for existing tables by comparing provided schema to existing schema

  11. Reference: Cassandra command options

    master

    The following options are available for the --cassandra command in the scalardb-schema-loader CLI.

    Options:
  -h, --host             Cassandra host IP (required)
  -P, --port             Cassandra Port
  -u, --user             Cassandra user (required)
  -p, --password         Cassandra password (required)
  -n, --network-strategy Cassandra network strategy (SimpleStrategy or NetworkTopologyStrategy)
  -c, --compaction-strategy Cassandra compaction strategy (LCS, STCS or TWCS)
  -R, --replication-factor Cassandra replication factor