CloudNativePG Helm Charts

repository·main·Indexed 20 days ago

https://github.com/cloudnative-pg/charts

Helm charts for deploying and managing CloudNativePG on Kubernetes. This includes charts for the CloudNativePG operator, the Barman Cloud plugin, and individual database clusters. It provides configuration for PostgreSQL database types (standard and PostGIS), operation modes (standalone, replica, and recovery), backup management via Barman, Prometheus monitoring, and PgBouncer poolers.

Tokens
20.6K
Snippets
56
Records
85
Agent score
65%

What's inside cloudnative-pg-charts

  1. Understand Cluster Operation Modes

    main

    The mode parameter determines how the Helm chart initializes the CloudNativePG cluster:

    • standalone (default): Creates a new CNPG cluster or updates an existing one.
    • replica: Creates a replica cluster from an existing CNPG cluster.
    • recovery: Creates a cluster from a backup, an object store, or via pg_basebackup.
  2. Resolve Critical Physical Replication Lag (>15s)

    main

    If replication lag exceeds 15 seconds, take the following actions:

    1. Terminate problematic transactions: Kill active queries running longer than 30 minutes that are not autovacuum processes.
    2. Scale Resources: Increase CPU and Memory. For Quality of Service (QoS) Guaranteed, set requests and limits to the same value in your Helm values.
    3. Enable WAL Compression: Reduces WAL file size and network bandwidth usage. This change does not require a restart.
    4. Upgrade Storage: Increase IOPS/throughput by migrating to a new StorageClass. Replace instances one by one, starting with standby replicas.
    5. Increase WAL Senders: For clusters with 9+ instances, ensure max_wal_senders is sufficient (should be $\ge$ number of instances).
    # Example: Scaling resources for QoS Guaranteed
    cluster:
      resources:
        requests:
          cpu: 4
          memory: 16Gi
        limits:
          cpu: 4
          memory: 16Gi
    
    # Example: Enabling WAL compression
    cluster:
      postgresql:
        parameters:
          wal_compression: "on"
    
    # Example: Increasing WAL senders
    cluster:
      postgresql:
        parameters:
          max_wal_senders: 15
  3. Handle High Transaction Volume in Logical Replication

    main

    If lag is caused by high transaction volume, consider these strategies:

    1. Batching: Break large transactions into smaller ones or use COPY instead of multiple INSERT statements.
    2. Row Filtering: Use ALTER PUBLICATION to only replicate necessary data or add table-level filters.
    3. Disable Triggers: Temporarily disable triggers on the subscriber for performance-critical periods (ensure you re-enable them).

    Example: Row Filtering

    -- Only replicate specific operations
    ALTER PUBLICATION publication_name SET (publish = 'insert, update, delete');
    
    -- Add table-level filtering
    ALTER PUBLICATION publication_name ADD TABLE table_name WHERE (condition);
  4. Resolve Permission and Data Type Mismatch Errors

    main

    Permission Issues

    Ensure the subscription owner has the necessary rights on the subscriber.

    1. Check Owner: SELECT usename, usesuper FROM pg_user WHERE usename = current_user;
    2. Grant Privileges:
      GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO subscription_user;
      GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO subscription_user;

    Data Type Mismatches

    If the schema between publisher and subscriber differs, replication may fail.

    1. Verify Schema: Compare information_schema.columns on both the publisher and subscriber.
    2. Fix Schema: Alter the subscriber table to match the publisher.
      ALTER TABLE table_name ALTER COLUMN column_name TYPE new_type;
  5. Resolve Receipt Lag (Network Issues)

    main

    Receipt lag indicates the subscriber is not receiving WAL messages quickly enough.

    Diagnosis:

    • Use ping to check latency between the subscriber and the PUBLISHER-HOSTNAME.
    • Use nc -zv to check connectivity to the publisher on port 5432.

    Resolution:

    1. Optimize Network Placement: Place clusters in the same region or availability zone.
    2. Tune PostgreSQL Parameters: Adjust the subscriber's configuration to be more aggressive with status updates.
    # In the subscriber's postgresql configuration
    postgresql:
      parameters:
        wal_sender_timeout: '60s'
        wal_receiver_status_interval: '10s'
    postgresql:
      parameters:
        wal_sender_timeout: '60s'
        wal_receiver_status_interval: '10s'
  6. Install the Barman Cloud CNPG-I plugin

    main

    Use the plugin-barman-cloud Helm chart to install the CNPG-I Barman Cloud Plugin.

    Refer to the Barman Cloud Plugin Chart documentation for installation instructions and advanced configuration settings.

  7. Uninstall the CloudNativePG operator

    main

    To remove the operator, use helm uninstall.

    CRITICAL WARNING: Uninstalling the chart does not remove the Custom Resource Definitions (CRDs). If you manually delete the CRDs, it will trigger a cascade deletion of every Cluster resource and all associated PostgreSQL data in their PVCs. This action is irreversible.

    helm uninstall cnpg --namespace cnpg-system
  8. Identify the type of logical replication lag

    main

    To determine if you are facing Receipt Lag, Apply Lag, or LSN Distance issues, connect to the subscriber cluster and run a query against pg_subscription and pg_stat_subscription. This will identify the primary_issue.

    kubectl exec -it svc/SUBSCRIBER-CLUSTER-rw -n NAMESPACE -- psql -c "
    SELECT
        s.subname,
        s.subenabled AS enabled,
        EXTRACT(EPOCH FROM (NOW() - ss.last_msg_receipt_time)) AS receipt_lag_seconds,
        EXTRACT(EPOCH FROM (NOW() - ss.latest_end_time)) AS apply_lag_seconds,
        COALESCE(pg_wal_lsn_diff(ss.received_lsn, ss.latest_end_lsn), 0) AS pending_bytes,
        CASE
            WHEN EXTRACT(EPOCH FROM (NOW() - ss.last_msg_receipt_time)) > 60 THEN 'High receipt lag'
            WHEN EXTRACT(EPOCH FROM (NOW() - ss.latest_end_time)) > 60 THEN 'High apply lag'
            WHEN COALESCE(pg_wal_lsn_diff(ss.received_lsn, ss.latest_end_lsn), 0) > 1024^3 THEN 'High LSN distance'
            ELSE 'Healthy'
        END as primary_issue
    FROM pg_subscription s
    LEFT JOIN pg_stat_subscription ss ON s.oid = ss.subid;
    "
  9. Resolve Initial Sync and Connection Errors

    main

    Initial Sync Errors (Missing Tables)

    If tables exist on the publisher but not the subscriber:

    1. Export Schema: pg_dump -h PUBLISHER-HOST -U postgres -s -t table_name database_name
    2. Import Schema: psql -h SUBSCRIBER-HOST -U postgres -d database_name < schema_dump.sql

    Connection/Timeout Issues

    If replication is failing due to network latency or connectivity:

    1. Test Connectivity: Run psql -h PUBLISHER-HOST -U postgres -d database_name -c "SELECT 1;" from the subscriber pod.
    2. Increase Timeouts: Adjust the subscription configuration parameters.
    # In subscription configuration
    spec:
      parameters:
        application_name: "my_subscription"
        synchronous_commit: "off"
  10. Enable and use the Console Pod for long-running tasks

    main

    The Cluster chart can deploy a dedicated console pod via a StatefulSet to execute long-running database tasks (like CREATE INDEX) that are resistant to network interruptions.

    To enable this feature, set the following Helm value: cluster.console.enabled=true.

    Important Considerations:

    • No PDB: There is no PodDisruption Budget for the console StatefulSet. Node maintenance may evict the pod and kill your session.
    • Ephemeral Tools: The pod has root access, allowing you to use apt install for additional tools. However, while the /root home folder is persisted, any installed tools will be lost if the pod restarts.
    • Startup Delay: Utilities are installed during pod startup; it may take a few seconds after a restart before they are available.
    cluster.console.enabled=true