DOCSEARCHbyAlgolia
Submit Docs

Vitess

repository·main·Indexed 12 days ago

https://github.com/vitessio/vitess

A cloud-native distributed database system built on top of MySQL that provides horizontal scalability through sharding. It enables applications to scale to massive sizes without requiring changes to query logic or manual data distribution management.

Tokens
162.6K
Snippets
511
Records
819
Agent score
97%

What's inside Vitess

  1. What is Vitess?

    main

    Vitess is a cloud-native, horizontally-scalable distributed database system built around MySQL. It enables unlimited scaling through generalized sharding while allowing application code and database queries to remain agnostic of how data is distributed across multiple database servers.

    Key capabilities include:

    • Generalized Sharding: Scale horizontally by splitting data across multiple nodes.
    • Transparent Data Distribution: Applications interact with Vitess as if it were a single database.
    • Dynamic Resharding: Split or merge shards as needed with an atomic cutover process that typically takes only a few seconds.

  3. Overview of Vitess Java modules

    main

    Vitess provides several Java-based modules for interacting with the Vitess cluster. When searching for these in Maven Central, note that all artifactIds are prefixed with vitess- (e.g., vitess-jdbc instead of jdbc).

    Available modules:

    • vitess-client: The core Java client library. It defines an interface for RPC systems but is agnostic of the underlying implementation. In open-source environments, it must be used in conjunction with vitess-grpc-client.
    • vitess-grpc-client: Provides the gRPC implementation for the vitess-client RPC interface.
    • vitess-jdbc: A JDBC driver implementation for Vitess.
    • vitess-hadoop: Provides Vitess support for Hadoop environments.
    • example: Contains usage examples for the client and JDBC modules.

  4. Understand the Vitess Go package structure

    main

    The Vitess Go codebase is organized into three main areas:

    1. General-purpose packages: Located at the top level of the go/ directory. These are designed to be used independently of Vitess.
    2. Vitess-specific packages: Located in the go/vt/ subdirectory. These contain logic specific to the Vitess ecosystem.
    3. Binaries: Located in the go/cmd/ subdirectory.

    For a complete list of packages and their specific purposes, refer to the official GoDoc documentation.

  5. Planned improvements for Vitess V3

    main

    The Vitess V3 project aims to evolve several core capabilities. Developers using Vitess should be aware of these upcoming functional areas which are currently in development or planned:

    • Transactions: Implementation of 2PC for cross-shard consistency and the use of MySQL SAVEPOINTs to ensure that partial work is rolled back if a request fails.
    • Resharding: Support for migrating tables between unsharded and sharded states (and vice versa) using tablet-type specific vschemas.
    • Post-processing: Enhanced SQL support including:
      • Joins: Trivial routing joins and cross-keyspace joins.
      • Aggregations & Group By: Support for COUNT, SUM, etc.
      • Order By: Merge-sort support for numeric and binary columns.
      • Clauses: Support for LIMIT and subqueries.
    • Live Vindex Creation: A workflow to create new lookup Vindexes for existing data, involving a specialized VTGate index for DML updates followed by a data backfill process.

  6. Use Vitess Bootstrap Images

    main

    Vitess bootstrap images are Docker images that contain the full environment expected after running bootstrap.sh and dev.env. These images are useful for providing a pre-configured environment for Vitess development or testing.

    Available image flavors include:

    • vitess/bootstrap:common: Contains dependencies common to all flavors.
    • vitess/bootstrap:mysql80: Bootstrap image for MySQL 8.0.
    • vitess/bootstrap:mysql84: Bootstrap image for MySQL 8.4.
    • vitess/bootstrap:percona80: Bootstrap image for Percona Server 8.0.

    Note: These images are not automatically rebuilt on every push to the Vitess main branch.

  7. Select the appropriate Vitess Docker image

    main

    Vitess provides different images depending on your requirements. All images include a specific MySQL/MariaDB version flavor. Note that on Docker Hub, only images with MySQL 5.7 are published to minimize maintenance overhead.

    ImageDescription
    liteRecommended. A stripped-down version of the base image. Automatically updated after every push to the main branch.
    baseContains all Vitess server binaries. Use this if you need a binary that is not included in the lite image.
    bootstrapA snapshot of the repository after running ./bootstrap.sh. Used to cache dependencies and avoid lengthy recompilations.

    Usage Summary:

    • For running Vitess: Use vitess/lite (preferably with a fixed version tag).
    • For latest binaries: Use the latest tag of vitess/lite.
    • For specific binaries not in lite: Use the base image.

  8. Understand the Vitess V3 API capabilities

    main

    The Vitess V3 API aims to make a sharded database look like a single database by removing the requirement for applications to be aware of the sharding key.

    Current V3 Capabilities:

    • Single-table SELECTs: Correctly routes queries to specific shards if the sharding column is provided (e.g., select a.col from a where a.id=:id).
    • IN clauses: Automatically breaks IN statements into shard-specific queries (e.g., select a.col from a where a.id in ::list).
    • Scatter queries: Sends queries to all shards and combines results if no specific shard can be determined (e.g., select a.col from a where a.col=:col).
    • Simple DMLs: Supports basic Data Manipulation Language operations.

    Planned V3 Capabilities (Next Phase):

    • Joins: Supports single-shard joins and cross-shard joins (where VTGate performs the join logic).
    • Subqueries: Supports correlated subqueries and subqueries in the FROM clause.
    • Aggregation: Supports single-shard aggregations (including those with joins/subqueries) and scatter aggregations if a unique vindex is present in the result columns.
    • Sorting: Supports ORDER BY by pushing down sorting to the grouped parts of the query.

  9. How query dependency resolution works in V3

    main

    In the V3 design, when a query is broken into route primitives (like joins), dependencies between these parts must be resolved. This is achieved using symbol tables generated during query analysis.

    To resolve an external reference (a reference to a symbol outside the current group):

    1. The external reference is converted to a corresponding bind var name.
    2. If the referenced column is not in the SELECT list of the target group, it is added.
    3. The column is marked with a column number to be exported as the chosen bind var name.

    Caveat: Aggregates cannot be added to a SELECT list during this step. Currently, this is safe because Aggregates cannot be pushed into joins, and external dependencies only occur within joins.

  10. How Vitess handles joins across different keyspaces using bind variables

    main

    When Vitess performs a join where tables reside on different keyspaces, it uses a bind variable mechanism to feed results from one query into another.

    Bind Variable Generation

    To join t1 and t2 on t2.id = t1.id, Vitess executes the first query and then uses the results to construct a second query using a generated bind variable.

    The bind variable name follows this functional algorithm: '_' + tableAlias + '_' + columnName

    Example: If the first query is select t1.a from t1, and the second query needs to filter by that value, Vitess generates a bind variable named :_t1_a.

    Best Practices for Query Stability

    To ensure predictable bind variable generation, avoid using non-standard column names (like expressions with special characters) in subqueries.

    Avoid:

    select * from (select a, count(*) from t1) t where t.count(*) > 0

    Use explicit aliases:

    select * from (select a, count(*) c from t1) t where t.c > 0

    This ensures Vitess can reliably build a bind variable like :_t_c for the outer query.

  11. Accessing symbol tables via the Select structure

    main

    Because subqueries can exist anywhere in the parse tree, symbol tables are stored directly within the AST to ensure they can be retrieved for dependency resolution.

    To implement this, the Select structure is extended with a Symbols (any) member. When a subquery analysis is completed, the Symbols member is set to point to that subquery's specific symbol table.

  12. Aggregate Operator Behavior

    main

    The Aggregate(Result, aggrExpressionList(columns)) -> Result operator groups rows based on the GROUP BY columns and applies aggregation functions to the aggregate columns.

    Key Behaviors:

    • Grouping: Any columns in the input Result that are not in the aggregate expression list are implicitly treated as part of the GROUP BY list.
    • Row Count: It produces no more rows than the input. If there are no GROUP BY columns, exactly one row is produced.
    • Column Count: It produces the same number of columns as the input.
    • Efficiency: Aggregate operations are highly efficient if the input rows are already sorted by the GROUP BY columns, as the operator can retain that sort order.
    • Special Case (Deduplication): If there are no aggregate columns, the operation acts as a deduplication (UNION).