DOCSEARCHbyAlgolia
Submit Docs

Zalando Postgres Operator

repository·master·Indexed 26 days ago

https://github.com/zalando/postgres-operator

A Kubernetes-native tool that automates the deployment and management of highly-available PostgreSQL clusters using Patroni. It supports PostgreSQL versions 14 through 18, providing capabilities for rolling minor updates, in-place major version upgrades, and Point-In-Time-Recovery (PITR) via WAL-G or WAL-E. The operator manages databases through Custom Resource Definitions (CRDs) and supports various cloud environments including AWS, GCS, and Azure.

Tokens
32.6K
Snippets
54
Records
167
Agent score
89%

What's inside zalando-postgres-operator

  1. Overview of Postgres Operator features

    master

    The Postgres Operator provides highly-available PostgreSQL clusters on Kubernetes using Patroni. It is managed via PostgreSQL manifests (CRDs), enabling infrastructure-as-code workflows without requiring direct Kubernetes API access for every operation.

    Key Operator Capabilities

    • Cluster Management: Rolling updates for minor versions, fast in-place major version upgrades (including global upgrades), and live volume resizing (AWS EBS, PVC) without pod restarts.
    • High Availability & Scaling: Streaming replication via Patroni, connection pooling with PGBouncer, and support for standby clusters from S3/GCS WAL archives or remote hosts.
    • Data Protection: Point-In-Time-Recovery (PITR) using pg_basebackup, WAL-G, or WAL-E. Supports logical backups to S3 or GCS.
    • Cloud & Environment Support: Native support for AWS, GCS, and Azure (including restore/cloning); also configurable for non-cloud environments. Compatible with OpenShift.
    • Security & Management: Basic credential/user management on K8s, custom TLS certificate support, and a UI for managing cluster manifests.

  2. Understand the Postgres Cluster manifest structure

    master

    Individual PostgreSQL clusters are defined using a Kubernetes custom resource (CRD) with the postgresql kind. The manifest is a YAML document where parameters and parameter groups are defined at the top level using camelCase.

    Important Precedence Rule: If a parameter is defined in both the operator's ConfigMap/CRD and the specific Postgres cluster manifest, the value provided in the cluster manifest takes precedence.

  3. Understand the scope of the Postgres Operator

    master

    The operator's primary scope is the provisioning, configuration, and cleanup of PostgreSQL clusters that use Patroni.

    Included in scope:

    • Provisioning and modifying Kubernetes resources.
    • Provisioning databases and roles once the cluster is running.
    • Orchestrating rolling updates to improve user experience.

    Not in scope:

    • Monitoring or tuning PostgreSQL performance. However, the operator supports globally configurable sidecars, allowing integration with tools like Prometheus, ZMON, or other Postgres-specific monitoring solutions.

  4. Understand the Postgres Operator core concepts

    master

    The Postgres Operator manages PostgreSQL clusters on Kubernetes by acting as a control loop that synchronizes the actual state of the cluster with the desired state defined in manifests.

    Key behaviors include:

    • Manifest Management: Watches for additions, updates, and deletions of PostgreSQL cluster manifests. When a new manifest is submitted, the operator spawns the cluster and necessary entities like K8s StatefulSets and Postgres roles.
    • Configuration Updates: Watches its own configuration (e.g., via a ConfigMap). If configuration changes (like a Docker image update), the operator performs rolling updates by re-spawning pods one-by-one.
    • State Synchronization: Periodically ensures the running cluster matches the desired state defined in the manifest.
    • Hands-free Operation: Designed for automated deploy pipelines where configuration is handled exclusively via manifests.

  5. Run an individual E2E test case

    master

    If you have already run the tests with NOCLEANUP=True, you can run specific test cases without recreating the cluster. Use the following pattern in the e2e directory:

    NOCLEANUP=True ./run.sh main <test_package_path>

    NOCLEANUP=True ./run.sh main tests.test_e2e.EndToEndTestCase.test_lazy_spilo_upgrade

  7. Configure initContainers for a PostgreSQL cluster

    master

    You can specify custom initContainers to run actions before the main PostgreSQL and sidecar containers start. The initContainers field accepts a full v1.Container definition.

    Requirement: The operator's global configuration must have enable_init_containers: true.

    spec:
  initContainers:
    - name: "container-name"
      image: "company/image:tag"
      env:
        - name: "ENV_VAR_NAME"
          value: "any-k8s-env-things"

  9. Restore a PostgreSQL cluster in place

    master

    To restore a database without changing the application's connection parameters, you can perform an in-place restore. This process is riskier as it involves deleting the existing database first.

    1. Ensure there is no writing activity on the DB.
    2. Save the current cluster's uid.
    3. Delete the existing postgresql Kubernetes resource.
    4. Deploy a new manifest with the same metadata.name, but set the spec.clone.cluster to the same name and provide the original uid and the desired timestamp.
    zkubectl delete postgresql acid-test-restore
    metadata:
  name: acid-minimal-cluster
# [...]
spec:
  # [...]
  clone:
    cluster: "acid-minimal-cluster"  # the same as metadata.name above!
    uid: "<original_UID>"
    timestamp: "2022-04-01T10:11:12.000+00:00"

  10. Set up the Go development environment

    master

    Postgres Operator requires Go 1.17 or later. The project follows the standard GOPATH structure. You should place the source code under ~/go/src/github.com/zalando/postgres-operator to ensure dependencies are resolved correctly.

    export GOPATH=~/go
mkdir -p ${GOPATH}/src/github.com/zalando/
cd ${GOPATH}/src/github.com/zalando/
git clone https://github.com/zalando/postgres-operator.git

  11. Configure TLS for the connection pooler

    master

    When a tls section is specified in the cluster manifest, the connection pooler (PgBouncer) will automatically use the same certificates.

    Important: The operator does not automatically sync the pooler deployment when TLS is added. To apply TLS settings to the pooler pods, you must toggle the enableConnectionPooler flag (e.g., set it to false and then back to true) to trigger a deployment update.

  12. Configure the Postgres Operator using CRD-based configuration

    master

    The recommended method for configuring the Postgres Operator is using the OperatorConfiguration Custom Resource Definition (CRD). This method is more powerful and supports nested YAML structures.

    Setup requirements:

    1. The operator deployment manifest must set the POSTGRES_OPERATOR_CONFIGURATION_OBJECT environment variable to the name of the postgresql-operator-configuration object in the operator's namespace.
    2. The operator must register the CRD during startup (controlled by enable_crd_registration, which defaults to true).

    Key characteristics:

    • Standard YAML: Non-scalar keys (lists/maps) use standard YAML syntax.
    • No built-in defaults: Unlike the ConfigMap method, parameters not supplied in the CRD receive an empty value. It is recommended to copy the default configuration manifest as a starting point.
    • Hierarchical structure: Configuration groups correspond to non-leaf keys in the YAML (e.g., the kubernetes group uses the kubernetes key).