DOCSEARCHbyAlgolia
Submit Docs

Percona Operator for MySQL

repository·main·Indexed 42 days ago

https://github.com/percona/percona-server-mysql-operator

Automates the lifecycle of highly available MySQL clusters on Kubernetes. Supports Group Replication (GA as of version 1.0.0) and Asynchronous Replication (Tech Preview), with integration for Percona Monitoring and Management (PMM), HAProxy, and MySQL Router.

Tokens
29.5K
Snippets
65
Records
122
Agent score
53%

What's inside Percona Operator for MySQL

  1. Overview of Percona Operator for MySQL

    main

    The Percona Operator for MySQL automates the deployment and management of highly available, enterprise-ready MySQL database clusters on Kubernetes. It is designed to follow best practices for fault-tolerant MySQL instances in both on-premises and cloud environments.

    Key Capabilities:

    • Cluster Deployment: Supports Group Replication MySQL clusters (with HAProxy or MySQL Router) and Asynchronous Replication MySQL clusters (with Orchestrator and HAProxy).
    • Service Exposure: Exposes clusters using standard Kubernetes Services.
    • Monitoring: Integrates with Percona Monitoring and Management (PMM).
    • Configuration: Allows for custom MySQL configuration and management of system user passwords.

  2. How the ClusterSet controller orchestrates operations

    main

    The PerconaServerMySQLClusterSet controller acts as an orchestrator over MySQL primitives. It does not manage the underlying MySQL data directly, meaning the loss of the Kubernetes cluster hosting the CR does not result in data loss; you can redeploy and re-apply the CR to resume management.

    Execution Model

    • Fast Verbs: The controller executes commands via kubectl exec into a dedicated mysqlshell-runner Pod.
    • Long-running Operations: Operations such as createReplicaCluster, setPrimaryCluster, and removeCluster are executed asynchronously via Kubernetes Jobs. This prevents the controller's reconciliation loop from being blocked by long-running tasks (like a multi-hour CLONE operation over a WAN).

    The mysqlshell-runner Pod

    To keep the main operator image lightweight and ensure version compatibility, the operator manages a separate mysqlshell-runner Pod for each ClusterSet CR. This runner pod is configured with a specific mysqlsh version that matches the target MySQL endpoints.

  3. How the mysqlshell-runner and ClusterSet Jobs Work

    main

    The operator uses two distinct mechanisms to execute MySQL Shell AdminAPI commands:

    1. mysqlshell-runner Deployment:

      • A long-lived, stateless Pod managed by the controller (named <cr-name>-mysqlshell-runner).
      • Used for standard AdminAPI calls via kubectl exec.
      • Does not use PVCs or baked-in Secrets; credentials are passed via stdin/env during invocation.

    2. Job-based Operations:

      • Used for long-running ClusterSet operations.
      • Jobs follow the naming pattern <cr-name>-<cluster-name>-<action> to prevent concurrent duplicate actions on the same cluster.
      • These jobs use the same image as the operator and execute commands by exec-ing into the mysqlshell-runner Pod.

  4. Understand the INVALIDATED state in InnoDB ClusterSet

    main

    In an InnoDB ClusterSet, if a forcePrimaryCluster() operation is performed while the old primary is unreachable, the old primary is marked as INVALIDATED in the metadata.

    • Why it happens: The old primary is fenced off because it may have accepted writes that the new primary does not have, preventing data divergence.
    • Recovery options:
      1. rejoinCluster(): A relatively cheap operation if the GTIDs (Global Transaction Identifiers) are compatible.
      2. Remove and Recreate: A slow operation that requires a full CLONE of the data from the current primary.

  5. Manage cross-cluster credentials with the clusterset user

    main

    For cross-cluster operations, the operator uses a dedicated MySQL user named clusterset. This user is provisioned on every operator-managed cluster during bootstrap and has the specific grants required to manage the mysql_innodb_cluster_metadata schema.

    To use the ClusterSet, you must provide a Kubernetes Secret containing the password for this clusterset user. The PerconaServerMySQLClusterSet CR references this via spec.credentialsSecret.

  6. Manage MySQL Shell runtime for ClusterSet

    main

    The Percona Operator does not bundle MySQL Shell (mysqlsh) within the operator image itself. Instead, it manages a dedicated mysqlshell-runner Pod for each ClusterSet Custom Resource (CR).

    To control which version of MySQL Shell is used for ClusterSet operations (such as the createReplicaCluster Job), you must specify the image via the spec.mysqlshellRunner.image field in your ClusterSet configuration.

    # Example configuration snippet
spec:
  mysqlshellRunner:
    image: mysql/mysql-shell:8.4.0

  7. How incremental backups work in Percona Server MySQL Operator

    main

    Incremental backups capture only the data pages changed since a previous backup, reducing time and storage compared to full backups. The operator maintains a backup chain following the pattern: full → inc1 → inc2 → ... → incN.

    Core Concepts

    • Log Sequence Numbers (LSN): Every InnoDB page has an LSN. An incremental backup copies only pages with an LSN newer than a reference LSN. The result is a set of .delta files.
    • Backup Chains: For a chain to be valid, each incremental's from_lsn must exactly match the prior backup's to_lsn found in the xtrabackup_checkpoints file.
    • Storage Layout: The operator uses an implicit storage convention to make chains discoverable. If a base backup is named cluster-ts-full, its incremental backups are stored under the prefix cluster-ts-full.incr/ using the naming convention <cluster>-<timestamp>-incr.
    • Streaming: Backups are streamed directly to cloud storage (S3, GCS, Azure) via xtrabackup --backup --stream=xbstream | xbcloud put. Because the sidecar is stateless and has no local copy of previous backups, the operator uses the --incremental-lsn=<LSN> flag to specify the starting point.

  8. Incremental backup storage and path conventions

    main

    The operator uses a specific path convention to logically link incremental backups to their base backup without requiring external metadata files. This allows for cross-cluster restores (backupSource) because the target cluster can discover the chain structure directly from the storage layout.

    Path Pattern:

    • Base Backup: <base-name>
    • Incremental Directory: <base-name>.incr/
    • Incremental Object Name: <cluster>-<timestamp>-incr

  9. Understand InnoDB ClusterSet cross-cluster replication

    main

    The Percona Operator for MySQL introduces support for InnoDB ClusterSet via a new namespaced Custom Resource Definition (CRD) called PerconaServerMySQLClusterSet (or ps-clusterset).

    An InnoDB ClusterSet is a high-level MySQL abstraction consisting of one PRIMARY InnoDB Cluster and one or more REPLICA InnoDB Clusters connected by asynchronous replication. Each REPLICA cluster is itself a fully functional Group Replication (GR) group. Only the PRIMARY cluster accepts writes.

    Key Capabilities

    • Cross-site replication: Manage replication between clusters located in the same Kubernetes cluster, different Kubernetes clusters, on-premises, or managed services.
    • Orchestration: A dedicated controller uses mysqlsh (MySQL Shell) to automate operations like bootstrapping, switchover, rejoining, and dissolving clusters.
    • Failover: Supports both planned switchovers and explicit emergency (force) failovers.

    Important Constraints

    • MySQL Version: Requires MySQL 8.0.27 or higher on all participating clusters.
    • MySQL Shell: Requires a matching mysqlsh version across all clusters.
    • Configuration: All clusters must have Group Replication (GR) compatible configurations and the CLONE plugin loaded.
    • Topology: InnoDB ClusterSet only supports Group Replication (GR) for the individual clusters; mixing async-mode (Orchestrator) and GR is not supported.

  10. Understand the PerconaServerMySQLClusterSet CRD

    main

    Cross-cluster replication is managed via a dedicated Custom Resource Definition (CRD) called PerconaServerMySQLClusterSet. Unlike individual MySQL clusters, a ClusterSet represents the entire replication topology across multiple clusters.

    Key Characteristics:

    • Topology-wide Invariants: A ClusterSet maintains exactly one writable primary Cluster and zero or more replica Clusters.
    • Network-only Coupling: The CRD references MySQL endpoints and credentials directly rather than referencing Kubernetes PerconaServerMySQL objects. This allows it to manage clusters located outside of Kubernetes (on-prem, managed services, or different K8s clusters).
    • Single Reconciliation Boundary: All topology-level operations (like switchovers or joining clusters) are serialized through this single CR to ensure consistency across the entire MySQL ClusterSet.

  11. Backup constraints and validation

    main

    The operator enforces several constraints to ensure backup integrity:

    • Storage Consistency: All backups in a single chain must use the same storageName. This is enforced during CR validation.
    • Base Requirement: An incremental backup cannot be created if no successful full backup exists for the same clusterName and storageName. The operator will reject the CR with an error: no succeeded full backup found for cluster X with storage Y.
    • Concurrency: The sidecar prevents concurrent backups on the same chain to avoid creating ambiguous chain forks (where two backups might share the same from_lsn).

  12. How InnoDB ClusterSet cross-site replication works

    main

    Cross-site replication is implemented using InnoDB ClusterSet and is only available for clusters using spec.mysql.clusterType: group-replication. It is not supported for async cluster types.

    Replication Roles

    • Primary Cluster: Accepts writes and replicates asynchronously to replica clusters. Its internal Group Replication (GR) membership remains unchanged.
    • Replica Cluster: Serves reads only (SUPER_READ_ONLY=ON). It receives asynchronous replication from the primary cluster's PRIMARY member and maintains its own internal GR for high availability within its local site.

    ClusterSet Lifecycle

    1. Bootstrap: A new replica cluster is initialized with spec.mysql.bootstrap.mode: manual so its first pod starts as a standalone instance rather than attempting to form a GR group immediately.
    2. Joining: The PerconaServerMySQLClusterSet controller runs createReplicaCluster(), which configures the standalone instance to join the ClusterSet.
    3. Scaling: Once the first member of the replica cluster is part of the ClusterSet, other pods in that site can join the local GR group using the standard addInstance path.

    To participate in a ClusterSet, a cluster must be an InnoDB Cluster or a clean standalone GR-capable instance.

    # Example of a cluster configured to become a replica in a ClusterSet
apiVersion: ps.percona.com/v1
kind: PerconaServerMySQL
metadata:
  name: cluster2
spec:
  mysql:
    clusterType: group-replication
    size: 3
    bootstrap:
      mode: manual                  # Required for ClusterSet replica clusters
  secretsName: cluster2-creds