DOCSEARCHbyAlgolia
Submit Docs

Postgres Flex

repository·master·Indexed 41 days ago

https://github.com/fly-apps/postgres-flex

Docker images and configurations for running highly available Postgres clusters on Fly.io. Supports horizontal scaling, TimescaleDB, and automated failover. Includes the flexctl CLI for Barman backup management, a monitoring service for zombie primary detection and dead member removal, and utilities for failover validation and event reporting.

Tokens
7.3K
Snippets
32
Records
45
Agent score
51%

What's inside Postgres Flex

  1. How primary fencing and zombie mode work

    master

    If a booting primary cannot be verified as the real primary, it is fenced to protect the cluster. The behavior depends on whether a real primary can be identified:

    If the real primary is resolvable

    1. The cluster is set to read-only.
    2. The real primary's IP is written to a zombie.lock file.
    3. The member's role is set to zombie.
    4. The member restarts. During boot, it reads the IP from zombie.lock and attempts to rejoin the original cluster.
    5. If successful, zombie.lock is cleared and the member boots as a standby.

    Note: Rejoining will not be attempted if the resolved primary is in a different region than the local PRIMARY_REGION environment variable. You must update PRIMARY_REGION to allow a rejoin.

    If the real primary is NOT resolvable

    1. The cluster is set to read-only.
    2. An empty zombie.lock file is created.
    3. Upon reboot, the empty file indicates a non-recoverable failure mode (e.g., unregistered members or complete member cycling).

  2. Understand disk capacity monitoring and read-only mode

    master

    The system monitors disk capacity at regular intervals. If disk usage exceeds a 90% threshold, all user-defined tables are automatically switched to read-only mode to prevent corruption or total failure.

    Read/write operations are automatically re-enabled once disk usage falls below the defined threshold (e.g., via file cleanup or volume extension).

  3. How the cluster verifies the real primary via quorum

    master

    To prevent split-brain scenarios, the cluster evaluates its state by checking connectivity to registered standbys within the primary region and querying them for the current primary's identity.

    Cluster State Dimensions

    • Total members: Total number of registered members (including the primary).
    • Total active members: Number of responsive members (always $\ge 1$).
    • Total inactive members: Number of registered members that are unresponsive.
    • Conflict map: A map[string]int that tracks how many standbys report a specific primary IP. For example, if two standbys report the same conflicting IP, the map is map[string]int{"<ip_address>": 2}.

    Quorum and Fencing

    The real primary is considered resolvable if a majority of members agree on its identity. Quorum is defined as: total_members_in_region / 2 + 1

    Important: Even if the booting primary meets quorum, it will be fenced if a conflict is detected. This protects against race conditions where an old primary recovers during an active failover.

  4. When cluster state is monitored for split-brain mitigation

    master

    To mitigate split-brain scenarios, the cluster evaluates its state during the following events:

    • On boot: Ensures a booting primary is not an old primary returning after a failure.
    • Standby connectivity events: Evaluates state during standby connect, reconnect, or disconnect events. If quorum is lost during these events, the primary is fenced.
    • Background monitoring: Regular interval checks act as a fallback for missed events.

  5. Calculate the split-brain detection window

    master

    For versions v0.0.36+, the split-brain detection window is the time it takes to identify a network partition and fence the primary.

    Detection Process

    1. Repmgr pings registered members (5s connect timeout).
    2. Repmgr waits up to 30s for standby reconnection before issuing a child_node_disconnect event.
    3. Cluster state is evaluated. If the primary is isolated, evaluation takes approximately 5s per standby.

    Formula

    Use the following formula to estimate the window: connect_timeout + standby_reconnect_timeout + (registered_standbys * 5)

    Example (3-node cluster)

    • Initial connect timeout: 5s
    • Standby reconnect timeout: 30s
    • Cluster evaluation: (2 standbys * 5s) = 10s
    • Total estimated window: 45s
    connect_timeout + standby_reconnect_timeout + (registered standbys * 5)

  6. Perform a manual failover in Postgres Flex

    master

    If automatic failover does not occur, you can manually promote a standby node to primary.

    Critical Requirement: The promotion candidate must reside within your PRIMARY_REGION.

    Workflow Summary

    1. Connect to the standby machine you wish to promote via fly ssh console.
    2. Verify the standby is healthy using repmgr node check (as the postgres user).
    3. Stop the current primary machine using fly machines stop <machine-id> --app <app-name>.
    4. Promote the standby using repmgr standby promote --siblings-follow. Always use the --siblings-follow flag to ensure other standbys reconfigure to follow the new primary.
    5. Restart the old primary using fly machines start <machine-id> --app <app-name>. It will automatically rejoin the cluster as a standby.
    # 1. Connect to the candidate
fly ssh console -s -a <app-name>

# 2. (Inside SSH) Verify health
su postgres
cd ~
repmgr node check

# 3. (New Terminal) Stop the current primary
fly machines stop <primary-machine-id> --app <app-name>

# 4. (Back to SSH) Promote the standby
repmgr standby promote --siblings-follow

# 5. (New Terminal) Restart the old primary
fly machines start <old-primary-machine-id> --app <app-name>

  7. Provision a 3-member High Availability Postgres cluster

    master

    To set up a highly available Postgres cluster on Fly.io, use the fly pg create command. It is recommended to provision at least 3 members within your primary region to ensure automatic failover works correctly. The primary region is determined by the environment variable defined in your fly.toml file.

    Before provisioning, ensure your flyctl is up to date.

    # Update flyctl first
fly version update

# Provision the cluster
fly pg create --name <app-name> --initial-cluster-size 3 --region ord

  8. Scale your Postgres cluster horizontally

    master

    You can scale your cluster by cloning existing machines into new regions. First, identify the active machines in your application, then use the fly machines clone command with a specific machine ID and target region.

    # List your active Machines
fly machines list --app <app-name>

# Clone a machine into a target region
fly machines clone <machine-id> --region <target-region>

  9. Update your Postgres Machines to the latest version

    master

    To ensure you have the latest features and bug fixes, you should periodically verify and update your running Postgres machines. Use fly image show to check your current version and fly image update to perform the upgrade.

    # Verify your current version
fly image show --app <app-name>

# Update your Machines to the latest version
fly image update --app <app-name>

  10. Resolve disk capacity issues by extending a volume

    master

    When your database enters read-only mode due to disk capacity, you must bring usage below 90%. The recommended method is to extend the existing Fly.io volume.

    Follow these steps:

    1. List your volumes to identify the correct Volume ID.
    2. Extend the volume to a larger size.
    3. Restart the Machine associated with that volume to expand the filesystem (FS) to the new size.
    # 1. List volumes
fly volumes list --app <your-app-name>

# 2. Extend the volume
fly volumes extend <volume-id> --size <new-size-gb> --app <your-app-name>

# 3. Restart the machine
fly machines restart <machine-id> --app <your-app-name>

  11. Monitor cluster state for zombie primary detection

    master

    The monitor_cluster_state logic (part of the monitor command) is used to ensure cluster health by detecting 'zombie' primary nodes.

    It performs the following lifecycle:

    1. Role Verification: It checks the local node's role via RepMgr. It only executes monitoring logic if the node is the PrimaryRoleName.
    2. Zombie Diagnosis: It runs flypg.PerformScreening to determine if the primary is a 'zombie' (a node that thinks it is primary but is actually disconnected or unhealthy).
    3. Quarantine: If ErrZombieDiscovered or ErrZombieDiagnosisUndecided is returned, the node is automatically quarantined using flypg.Quarantine to prevent split-brain scenarios.
    4. Recovery: If no zombie is detected but a ZombieLock exists, the monitor clears the lock via flypg.RemoveZombieLock and broadcasts a state change to enable read/write access via flypg.BroadcastReadonlyChange(ctx, node, false).

  12. Understand the backup schedule monitoring logic

    master

    The backup monitor ensures that full backups are performed according to a defined schedule. It operates based on the following logic:

    1. Primary Node Check: Only the current PostgreSQL Primary node performs backups. The monitor continuously checks the node's primary status to handle failovers correctly.
    2. Backup Frequency: The interval between full backups is determined by the FullBackupFrequency setting in the Barman configuration. If not explicitly set, it uses a default schedule.
    3. Initial Backup: If no previous backups are detected (last backup time is zero) and the node is the Primary, the monitor triggers an immediate initial base backup.
    4. Scheduling: The monitor calculates the time remaining until the next backup is due. If the calculated time is less than zero, a full backup is triggered.
    5. Retry Mechanism: If a backup fails, the system attempts to retry the operation with a 30-second interval (backupRetryInterval) for up to 10 attempts before returning an error.
    6. Failover Resilience: Even non-primary nodes run the monitor to ensure that as soon as a node is promoted to Primary, it is ready to resume the backup schedule immediately.