DOCSEARCHbyAlgolia
Submit Docs

Apache ShardingSphere

repository·master·Indexed 12 days ago

https://github.com/apache/shardingsphere

An enterprise distributed database ecosystem providing a standardized enhancement layer for heterogeneous databases. It supports distributed computing, data security, and unified governance via a lightweight JDBC driver and a standalone database proxy. Key features include data sharding, read/write splitting, distributed transactions, and an observability framework called ShardingSphere-Agent that supports Prometheus and OpenTelemetry.

Tokens
282.7K
Snippets
619
Records
1K
Agent score
92%

What's inside ShardingSphere

  1. Overview of Chaos Mesh for Kubernetes

    master

    Chaos Mesh is an open-source chaos engineering platform designed for Kubernetes. It uses Kubernetes Custom Resource Definitions (CRDs) to orchestrate failure scenarios and simulate faults in production or testing environments. It consists of three primary components:

    • Chaos Dashboard: A WebUI for designing, monitoring, and managing chaos scenarios and RBAC permits.
    • Chaos Controller Manager: The core logic component that schedules designed Chaos CRs using various controllers (e.g., PodChaos Controller, WorkerFlow Controller).
    • Chaos Daemon: The execution component running as a DaemonSet. It typically requires privileged access to interfere with network equipment, file systems, and kernels within target Pod Namespaces.

  2. Overview of ShardingSphere on Cloud deployment options

    master

    ShardingSphere on Cloud provides automated deployment and maintenance solutions for various cloud environments to reduce manual labor. Available deployment methods include:

    • Kubernetes:
      • Helm Charts: For one-click deployment of ShardingSphere-Proxy.
      • Operator: For one-click deployment and automated maintenance.
    • AWS:
      • CloudFormation: For rapid deployment using Stack templates.
      • Terraform: For rapid deployment using Terraform scripts.

    These solutions support advanced features like automatic horizontal scaling, high availability, observability, and security compliance.

  3. What is Apache ShardingSphere?

    master

    Apache ShardingSphere is a Database Plus ecosystem designed to sit on top of heterogeneous databases. It acts as a database operating system that maximizes the capabilities of existing databases by providing a unified data access entry point and enhanced distributed computing capabilities.

    Its core pillars are:

    • Connection: Adapts database protocols, SQL dialects, and storage formats to connect applications to multi-mode heterogeneous databases.
    • Enhancement: Provides distributed computing (sharding, read-write splitting, federated queries), data security (encryption, masking, auditing), traffic control (circuit breaking, rate limiting), and observability.
    • Pluggability: Uses a micro-kernel + three-layer pluggable architecture to decouple the kernel, functional components, and ecosystem integrations.

  4. Overview of Apache ShardingSphere Data Encryption

    master

    Apache ShardingSphere provides a transparent data encryption solution designed to protect sensitive information (such as ID numbers, phone numbers, and card numbers) within a database.

    Key benefits include:

    • Transparency: Encryption and decryption are handled by ShardingSphere, allowing businesses to comply with security regulations without modifying existing business logic or SQL statements.
    • Low Risk: It avoids the need to reconstruct or modify complex encryption systems when security requirements change.
    • Compliance: Enables quick implementation of compliant data storage for new services requiring sensitive data protection.

  5. Overview of Chaos Blade for Cloud-Native High Availability

    master

    Chaos Blade is an open-source chaos engineering project by Alibaba designed to solve high availability issues in cloud-native processes.

    Key features include:

    • Extensive Coverage: Supports three major platforms, four computing languages, over 200 experimental scenarios, and over 3000 parameters for fine-grained control.
    • ChaosBlade-Box: A management platform that handles experimental tools. It manages Chaos Blade and can aggregate experimental tools from other platforms like Litmuschaos.

  6. Overview of ShardingSphere Agent

    master

    ShardingSphere-Agent is an observable framework for ShardingSphere implemented as a Java Agent. It provides observability through three main plugin types:

    • Logging: Records ShardingSphere logs (supports File output).
    • Metrics: Collects and exposes monitoring metrics (supports Prometheus).
    • Tracing: Obtains link trace information for SQL parsing and SQL execution (supports OpenTelemetry).

  7. Overview of ShardingSphere-Proxy

    master

    ShardingSphere-Proxy is a transparent database proxy that encapsulates database binary protocols (such as MySQL and PostgreSQL) to support heterogeneous languages. It acts as a database server, allowing DBAs to manage and retrieve data using any standard terminal compatible with the supported protocols (e.g., MySQL Command Client, MySQL Workbench).

    Key Characteristics:

    • Transparency: It is totally transparent to applications; applications can connect to it as if it were a standard MySQL/PostgreSQL instance.
    • Architecture:
      • Frontend: Handles client communication using an NIO framework (adaptive to Epoll on Linux). It manages the encoding/decoding of the database protocol.
      • Core-module: Parses, rewrites, routes, and conflates SQL using Sharding-Core.
      • Backend: Interacts with real databases using a Hikari connection pool (currently using BIO, with NIO support planned).

  8. Performance Optimizations in ShardingSphere 5.1.0

    master

    Starting with version 5.1.0, Apache ShardingSphere introduced several kernel-level optimizations to improve throughput and concurrent performance, particularly for ShardingSphere-Proxy. Key optimizations include:

    • Correcting java.util.Optional usage: Replacing orElse(T other) with orElseGet(Supplier<? extends T> other) to prevent unnecessary execution of method calls or modification operations when the value is already present.
    • Optimizing ConcurrentHashMap.computeIfAbsent: In Java 8 environments, frequent calls to computeIfAbsent for the same key can cause contention due to internal synchronization. ShardingSphere now uses a double-checked pattern (get followed by computeIfAbsent) to avoid this.
    • Reducing java.util.Properties overhead: Moving frequent getProperty calls from hot execution paths (like sharding algorithms) to the init phase to avoid synchronization overhead during runtime.
    • Removing Collections.synchronizedMap: Replacing synchronized maps with standard maps in read-heavy scenarios (e.g., TableMetaData) where modifications only occur during the initial phase.
    • Replacing String.format with concatenation: Using direct string concatenation (+) instead of String.format for simple delimiter wrapping to reduce CPU overhead.

  9. Benefits of using Apache ShardingSphere

    master

    Based on the Zhongshang Huimin enterprise use case, Apache ShardingSphere provides the following advantages for distributed database management:

    1. Performance Improvement: By rebuilding architecture to control single-table data volume, it can significantly reduce slow SQL queries (by nearly 50% in the reported case).
    2. Reduced R&D Costs: It eliminates the need to develop custom sharding components from scratch, lowering development risks and resource expenditure.
    3. Scalability: It provides built-in support for advanced features such as data encryption, distributed transactions, and shadow libraries (for testing/shadowing), allowing the system to grow with evolving requirements.

  10. Case Study: Zhongshang Huimin's Transaction Middle Platform Architecture

    master

    This case study describes how Huimin (an FMCG service provider) transitioned from a PHP stack to a Java-based microservices architecture to build a 'transaction middle platform'. The architecture focuses on reducing coupling, increasing scalability, and ensuring low latency by extracting business commonalities across order management, fulfillment, and cost calculation modules.

    Key architectural strategies used during the rebuilding process include:

    • System Splitting: Dividing responsibilities according to the order lifecycle to ensure independent systems with exclusive responsibilities.
    • Data Splitting: Separating data concerns to support business growth.
    • Read/Write Splitting: Ensuring core function stability by separating read and write operations.
    • Logic Decoupling: Minimizing the influence of different business lines on each other to reduce complexity.
    • External Indexing: Using ElasticSearch to solve external index problems and reduce database index pressure.

  11. Understanding Apache ShardingSphere's core value and features

    master

    Apache ShardingSphere is a pluggable platform designed to manage data sharding, allowing developers to interact with a sharded database cluster as if it were a single, standalone database. It addresses the scalability limitations of relational databases by acting as a middleware that distributes queries across multiple databases.

    Key features include:

    • Versatility: Supports any database meeting the SQL-92 standard. It includes a complete SQL engine for accurate routing and provides three types of distributed transactions (including strongly consistent and flexible transactions).
    • Efficiency: Offers two access modes: ShardingSphere-JDBC (runs in the same thread as the Java application for high performance) and ShardingSphere-Proxy (supports applications in any programming language via MySQL or PostgreSQL protocols).
    • Pluggability: Built using Java SPI (Service Provider Interface), allowing users to easily add or remove databases, functions, and sharding algorithms.
    • User-friendliness: Hides the complexity of distributed transactions and sharding logic. It provides DistSQL and AutoTable strategies to simplify configuration for DBAs.

  12. What is the Database Plus concept in ShardingSphere

    master
    "Database Plus" is ShardingSphere's positioning and development concept. Instead of creating a new type of relational database from scratch (like NewSQL), ShardingSphere acts as an enhanced computing engine and ecosystem layer above existing relational databases. It aims to improve the scalability and efficiency of relational databases in distributed scenarios, helping them handle high concurrency and large-scale data sharding while leveraging their existing mature transaction capabilities.