DOCSEARCHbyAlgolia
Submit Docs

TiDB Distributed SQL Database

repository·master·Indexed 12 days ago

https://github.com/pingcap/tidb

An open-source, cloud-native, distributed SQL database that is MySQL-compatible, featuring an HTAP architecture to handle both OLTP and OLAP workloads. The ecosystem includes tools such as BR for backup and restore, BenchDB for performance testing, Importer for data generation, Dumpling for SQL dumps, and TiDB-Lightning for high-performance TB-scale data imports.

Tokens
179.5K
Snippets
400
Records
771
Agent score
97%

What's inside TiDB

  1. Overview of TiDB Amazon EBS Backup & Restore

    master

    TiDB provides a block-level backup and recovery solution leveraging Amazon EBS snapshots. This method is designed to minimize the impact on the live TiDB cluster (targeting < 5% impact on QPS and transaction latency) and significantly shorten backup and restore times compared to traditional methods.

    Key Goals:

    • Minimize Cluster Impact: QPS/Transaction Latency < 5%.
    • Fast Backup: Target < 1 hour.
    • Fast Restore: Target < 2 hours.

    Limitations:

    • Cloud Provider: Only supports AWS EBS. Does not support EFS, local instance stores, or other cloud providers.
    • Component Support: Does not support TiFlash backup & restore.

  2. What is TiDB and its key features?

    master

    TiDB is an open-source, cloud-native, distributed SQL database designed for high availability, horizontal and vertical scalability, strong consistency, and high performance.

    Key capabilities include:

    • Distributed Transactions: Uses a two-phase commit protocol for ACID compliance and strong consistency across multiple nodes.
    • Scalability: Supports horizontal scaling (adding nodes) and vertical scaling (increasing resources) without downtime by separating computing from storage.
    • High Availability: Uses the Raft consensus protocol for reliability and automated failover, ensuring data is written to a majority of replicas.
    • HTAP (Hybrid Transactional/Analytical Processing): Provides both a row-based engine (TiKV) and a columnar engine (TiFlash) for optimized transactional and analytical workloads.
    • MySQL Compatibility: Compatible with MySQL 8.0, allowing the use of existing MySQL protocols, frameworks, and tools with minimal code changes.
    • Cloud-Native: Deployable on public clouds, on-premises, or via Kubernetes using the TiDB Operator.

  3. Introduction to the Timer framework

    master

    The Timer framework is an internal TiDB framework designed to run tasks periodically with specific scheduling semantics. It is suitable for scenarios requiring:

    • Periodic tasks with specific semantics: e.g., running a task every 5 minutes or every day at a specific time (00:00:00).
    • Distributed execution: Ensuring a task runs on only one node in a distributed environment (e.g., a 3-node TiDB cluster) and supporting recovery during process restarts.
    • Persistence and Monitoring: Accessing persisted schedule runtime information for monitoring purposes.
    • Message Delivery Semantics: Trigger actions that require at-least-once, at-most-once, or exactly-once delivery guarantees.

    When NOT to use the Timer framework:

    • High-frequency tasks: For tasks requiring very high frequency (e.g., every 1 second), the framework is inefficient due to the overhead of calls to the store during every trigger. Use standard Go time.Timer instead.
    • Heavy jobs: The framework is designed to schedule a lightweight "trigger" action. It is not a distributed job scheduler for heavy workloads. For heavy jobs, use a distributed job scheduler like disttask.

  4. What is Dumpling?

    master

    Dumpling is a tool and a Go library designed to create SQL dumps from MySQL-compatible databases. It is specifically intended as a replacement for mysqldump and mydumper when working with TiDB.

    Key features include:

    • Splitting SQL dumps into multiple files for easier management.
    • Parallel export of multiple tables to increase execution speed.
    • Support for multiple output formats (e.g., SQL, CSV).
    • Native support for writing directly to cloud storage like S3 and GCS.
    • Advanced table filtering capabilities.

  5. Overview of the MySQL Compatible SQL Parser

    master

    The parser project is a high-performance Golang-based SQL parser designed to be fully compatible with MySQL syntax. It is built using goyacc in a bottom-up approach, making it efficient at constructing Abstract Syntax Trees (AST) via a state machine.

    Key characteristics include:

    • High MySQL Compatibility: Supports almost all MySQL features (defined in parser.y and hintparser.y).
    • Extensibility: New syntax can be added with minimal changes to Yacc and Golang code.
    • Performance: Optimized for building AST trees efficiently.

  7. Overview of TiDB-Lightning

    master
    TiDB-Lightning is a high-performance data import tool designed to import data at TB scale into TiDB clusters. It supports different import modes, including Physical Import Mode, which can be utilized via the IMPORT INTO SQL statement to import data files or results from a SELECT statement into an empty table in TiDB.

  8. Understand TiDB Log-based Incremental Backup

    master

    TiDB Log-based Incremental Backup is a solution designed to back up incremental transactional data directly from the TiKV server. It is intended to work alongside full backups to minimize Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

    Key Capabilities:

    • Point in Time Recovery (PITR): Supports restoring data to a specific timestamp.
    • Scalability: Designed to support large clusters (e.g., 200T TiDB clusters with 66 TiKVs).
    • Low Impact: The log-based backup process is designed to have nearly non-existent impact on the running TiDB cluster.
    • Granularity: Supports both cluster-level and table-level backups.
    • Security & Efficiency: Supports data encryption and compression.

    Target Performance Goals:

    • RTO (Recovery Time Objective): < 6 hours.
    • RPO (Recovery Point Objective): < 5 minutes.

  9. Use the Column Mapping library for table transformations

    master

    The column-mapping library provides a unified way to map table columns during data operations. It is primarily used to:

    • Add a prefix to char, varchar, or text columns.
    • Add a suffix to char, varchar, or text columns.
    • Generate a partition ID for sharding schemas or tables (specifically for int64 ID columns).

    Note: The library currently does not change column types or table structures.

  10. Overview of TiDB Built-in SQL Diagnostics

    master

    TiDB is introducing built-in SQL diagnostics to address the fragmentation of existing diagnostic methods. Traditionally, diagnosing TiDB clusters required switching between external Linux tools (e.g., perf, iosnoop, iotop, vmstat, sar), monitoring systems (Prometheus/Grafana), log files, HTTP APIs, and various system tables.

    This built-in diagnostic feature aims to centralize information by exposing diagnostic data through system tables, allowing users to query cluster-wide diagnostic information directly using standard SQL. This approach is designed to simplify operations, improve the ability to detect issues early, and facilitate faster troubleshooting and recovery across the entire cluster.

  11. What is the Extension Authentication Plugin?

    master
    The Extension Authentication Plugin is a feature in TiDB that allows users to implement custom authentication and authorization logic. By providing a plugin, users can integrate TiDB with their own existing authentication systems instead of relying solely on TiDB's built-in mechanisms. This feature is designed to improve TiDB's compatibility with MySQL's authentication plugin architecture.

  12. What is a TTL Table in TiDB

    master

    A TTL (Time-To-Live) table is a table that automatically deletes expired rows based on a time threshold. This is useful for managing transient data, such as verification codes or session logs.

    To implement TTL, the table must include a column of type DATE, DATETIME, or TIMESTAMP. TiDB compares the value in this column with the current time; if the difference exceeds a predefined threshold, the row is automatically marked for deletion.