MOCO

MOCO is a MySQL operator for Kubernetes designed to manage MySQL clusters using GTID-based semi-synchronous replication.

Key characteristics include:

• Compatibility: Uses standard MySQL replication instead of Group Replication to avoid specific limitations.
• Safety: Ensures writes only occur on a single primary instance at a time and configures loss-less semi-synchronous replication with sufficient replicas. It also detects and excludes instances with errant transactions.
• Availability: Supports fast primary switching during failures or restarts and allows clusters of up to 5 instances.

MOCO manages clusters of mysqld instances using the MySQLCluster custom resource.

• Cluster Size: A cluster can consist of 1, 3, or 5 instances.
• Replication: For clusters of size 3 or 5, MOCO configures semi-synchronous GTID-based replication.
• Roles:
  • Primary: The single writable instance and the source of truth for replication.
  • Replica: Read-only instances that replicate data from the primary.
• Fault Tolerance:
  • A 3-instance cluster can tolerate up to 1 replica failure.
  • A 5-instance cluster can tolerate up to 2 replica failures.

MOCO manages a primary-replica style clustering architecture for MySQL instances on Kubernetes.

Key characteristics of this model include:

• Primary Instance: The single instance allowed to perform write operations.
• Replica Instances: Read-only instances that replicate data from the primary.
• Replication Method: Uses GTID-based replication. MOCO does not support non-GTID based replications.
• Failover & Selection: Supports automatic primary selection, switchover, and failover.
• Consistency: Designed to prevent split-brain scenarios and supports all four transaction isolation levels.
• External Replication: Supports replication from an external MySQL instance into the cluster.

MOCO manages MySQLCluster pods using Kubernetes StatefulSets. To prevent a rolling update from proceeding if the MySQLCluster becomes unhealthy, MOCO uses StatefulSet partitions to control the rollout.

By default, MOCO sets the StatefulSet partition to the same value as the number of replicas. This ensures that pods are updated one by one (starting from the highest pod index) only when the cluster is healthy. As each pod update succeeds and the cluster remains healthy, MOCO decrements the partition value until it reaches 0, completing the rollout.

MOCO manages backup and restore operations using standard Kubernetes resources:

• Backup (CronJob): When backup is enabled for a MySQLCluster, MOCO creates a CronJob. If backup is subsequently disabled, the CronJob is deleted.
• Restore (Job): To perform a data restoration, MOCO creates a Job.
  • If the Job finishes successfully, MOCO deletes it.
  • If the Job fails, MOCO leaves the Job in the cluster for debugging.

The MySQLClusterStatus provides the observed state of the cluster, including:

• currentPrimaryIndex: The index of the current primary Pod in the StatefulSet.
• syncedReplicas: Number of instances (including primary) that are currently synced.
• errantReplicas: Number of instances with errant transactions.
• backup: The BackupStatus of the last successful backup.
• reconcileInfo: Version information for the operator reconciler.

BackupStatus details: Contains metadata about the last successful backup, such as time, elapsed duration, sourceUUID, gtidSet, and the sizes of the dumpSize (full dump) and binlogSize (binlog tarball) in bytes.

A BackupPolicy is a namespaced Kubernetes Custom Resource used to define periodic backup schedules and configurations. It should be referenced from a MySQLCluster resource.

The policy is composed of a spec which defines the schedule and the job execution parameters.

apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  name: my-backup-policy
  namespace: default
spec:
  schedule: "0 2 * * *"
  jobConfig:
    serviceAccountName: moco-backup-sa
    bucketConfig:
      bucketName: my-backup-bucket
    workVolume:
      ephemeral:
        size: 10Gi

MOCO automatically prepares three sets of users with specific privileges:

1. moco-readonly: Can read all tables of all databases. Privileges include PROCESS, REPLICATION CLIENT, REPLICATION SLAVE, SELECT, SHOW DATABASES, and SHOW VIEW.
2. moco-writable: Can create users, databases, or tables. Includes ALTER, CREATE, DELETE, DROP, INSERT, UPDATE, etc. Note: moco-writable cannot edit tables in the mysql database.
3. moco-admin: The super user.

You can use moco-writable or moco-admin to create additional users and grant them privileges.

$ kubectl moco mysql -u moco-writable test -- -e "CREATE USER 'foo'@'%' IDENTIFIED BY 'bar'"
$ kubectl moco mysql -u moco-writable test -- -e "CREATE DATABASE db1"
$ kubectl moco mysql -u moco-writable test -- -e "GRANT ALL ON db1.* TO 'foo'@'%"

Installing or updating CRDs

By default, the Helm chart installs and updates Custom Resource Definitions (CRDs). If you prefer to manage CRDs manually, set crds.enabled: false in your values.

Removing CRDs

Warning: Helm will NOT remove CRDs during an uninstallation because they are annotated with helm.sh/resource-policy: keep. To fully remove MOCO, you must delete the CRDs manually using kubectl delete crd <crd-name>.

Migration to v0.11.0 or higher

When upgrading from chart v0.10.x or lower, you MUST ensure crds.enabled is set to true (the default). Setting this to false during an upgrade will cause the CRDs to be removed, leading to permanent data loss.