DOCSEARCHbyAlgolia
Submit Docs

sql-splitter Documentation

website·Indexed 31 days ago

https://www.sql-splitter.dev/

A high-performance command-line toolkit for working with SQL dump files. It allows users to split, merge, convert, validate, sample, shard, diff, redact, and query dumps across MySQL, PostgreSQL, SQLite, and MSSQL dialects. Features include a streaming architecture for large files, synthetic data generation, and AI-driven workflow optimization.

Tokens
67K
Snippets
355
Records
646
Agent score
45%

What's inside sql-splitter

  1. Overview of sql-splitter capabilities

    sql-splitter is a dump-first, CLI-first tool designed for automation and CI/CD pipelines. It operates directly on SQL dump files on disk (including compressed files) rather than live database connections.

    Key technical specifications:

    • Performance: Approximately 1 GB/s end-to-end split throughput.
    • Memory Footprint: Low memory usage (20–150 MB), utilizing a streaming architecture to handle files larger than available RAM.
    • Dialect Support: Supports MySQL, PostgreSQL, SQLite, and MSSQL, including specific features like COPY FROM stdin and GO batches.
    • Compression Support: Automatically detects gzip, bzip2, xz, and zstd compression.

  2. Overview of sql-splitter capabilities

    sql-splitter is a command-line tool designed for high-performance manipulation of SQL dump files. It uses a streaming architecture to handle multi-GB dumps with constant memory usage. Key use cases include automation via JSON output and glob patterns, CI/CD pipeline integration for validation and integrity checks, and database migrations between MySQL, PostgreSQL, SQLite, and MSSQL.

  3. Overview of Planners for multi-column and cross-table invariants

    Planners are used when independent generators might produce individually valid values that violate a shared equation, ordering, state transition, or relationship. A planner owns several columns as a single unit.

    Key constraints:

    • Planner-owned columns must not have their own separate column generators.
    • The compiler resolves reads/writes, rejects ownership conflicts, and orders planners based on dependencies.
    • Row counts are handled separately; for example, relation.children (a row-count rule) determines how many child rows exist, while a planner manages the values within those rows.
    tables:
  sessions:
    planners:
      - kind: temporal.interval
        columns:
          { start: started_at, end: ended_at, duration: duration_seconds }
        start: { kind: range, min: 2025-01-01, max: 2025-12-31 }
        duration: { kind: uniform, unit: seconds, min: 60, max: 3600 }

  4. Understand and use Modifiers for post-generation transformations

    Modifiers transform the base value produced by a generator. They are executed in the order they are listed in the configuration; changing the order can affect the final value's nullability, uniqueness, length, and formatting. All built-in modifiers operate only on their own column, stream row by row, and are deterministic under a fixed seed.
    generator: { kind: internet.email }
modifiers:
  - { kind: case, mode: lower }
  - { kind: prefix, value: "test+" }
  - { kind: truncate, max_length: 255 }
  - { kind: unique, max_attempts: 20 }

  5. Supported SQL Dialects in sql-splitter

    sql-splitter supports four SQL dialects. For each dialect, it recognizes specific detection signals (markers) in the first 8 KB of the file to identify the format. If a file is ambiguous, it defaults to MySQL.

  6. Understand multi-tenant database concepts in sql-splitter

    sql-splitter provides tools for managing multi-tenant architectures:

    • Tenant: A customer or organization whose data is logically separated from others.
    • Multi-Tenant: An architecture where one database serves multiple tenants, usually isolated by a tenant identifier column.
    • Sharding: In the context of sql-splitter, this refers to extracting a subset of data belonging to a specific tenant from a database dump (implemented via the shard command).
    • Tenant Column: The column (e.g., tenant_id, org_id, company_id, account_id) that identifies the owner of a row. sql-splitter can auto-detect these common names.

  7. Understand sql-splitter automated release workflows

    Releases are automated via GitHub Actions. Pushing a version tag triggers a workflow that:

    1. Builds binaries for multiple platforms: Linux (x86_64, ARM64), macOS (Intel, Apple Silicon), and Windows (x86_64).
    2. Creates a GitHub Release containing these binaries.
    3. Publishes the crate to crates.io using the CARGO_REGISTRY_TOKEN repository secret via cargo publish.

  10. Generate ERD diagrams from SQL dumps using the graph command

    The graph command (alias gr) parses CREATE TABLE statements and foreign key definitions to build a dependency graph where nodes represent tables and edges represent foreign key relationships (Referencing Table $\to$ Referenced Table). This is useful for schema documentation, dependency analysis, and planning safe DROP operations.
    sql-splitter graph <INPUT> [OPTIONS]
# Example using alias
sql-splitter gr dump.sql -o schema.html

  11. sql-splitter project structure overview

    The project is organized into the following key directories:

    • src/: Source code
      • main.rs: Entry point
      • cmd/: CLI command implementations (split, merge, analyze, etc.)
      • parser/: SQL parsing logic
      • splitter/, merger/, analyzer/: Core logic for splitting, merging, and analysis
      • convert/, validate/, sample/, shard/, differ/, redactor/: Specialized SQL processing modules
      • graph/: ERD and graph logic
      • duckdb/: DuckDB integration
      • writer/: Output writers
    • tests/: Test modules and fixtures/ (test SQL files)
    • man/: Man pages
    • website/: Documentation site

  12. Understand "Table has no primary key" warnings

    Warnings about missing primary keys (PKs) occur because PK duplicate detection requires a primary key to function. For tables without PKs, sql-splitter only performs basic syntax and encoding checks. These warnings are informational unless the --strict flag is used, which treats these warnings as errors.