DOCSEARCHbyAlgolia
Submit Docs

pg_auto_failover

repository·main·Indexed 23 days ago

https://github.com/hapostgres/pg_auto_failover

A PostgreSQL extension and service providing automated high availability (HA) through a monitor-based state machine. It manages primary and secondary nodes using synchronous replication to ensure data correctness and safe failover. Supports PostgreSQL versions 13 through 18 and includes the pg_autoctl utility for node operation and supervision.

Tokens
98.3K
Snippets
193
Records
502
Agent score
77%

What's inside pg_auto_failover

  1. What is pg_auto_failover?

    main

    pg_auto_failover is an extension and service for PostgreSQL designed to monitor and manage automated failover for a Postgres cluster. It is optimized for simplicity and correctness and supports PostgreSQL versions 13 through 18.

    It consists of three primary components:

    1. pgautofailover: A PostgreSQL extension.
    2. Monitor: A PostgreSQL service that operates the pg_auto_failover monitor.
    3. Keeper: A component used to operate your PostgreSQL instances, managed via the pg_autoctl run command.

  2. What is pgaftest and how to use its modes

    main

    pgaftest is the pg_auto_failover integration test runner. It uses .pgaf specification files to describe cluster topologies and test steps, then orchestrates the cluster using Docker Compose.

    It supports two primary modes of operation:

    1. CI mode (pgaftest run): A headless mode that produces TAP output and exits with a non-zero status on failure. Ideal for automated pipelines.
    2. Interactive mode (pgaftest cluster setup or pgaftest tmux): Keeps the cluster running and provides a shell or tmux session for manual exploration and debugging.

  3. Overview of pg_autoctl commands

    main
    The pg_autoctl tool is the primary command-line interface for managing pg_auto_failover clusters. It uses a hierarchical command structure where specific tasks are handled by sub-commands. Each sub-command has its own dedicated manual page for detailed flag and argument definitions.

  5. Use pg_autoctl inspect for read-only diagnostics

    main

    The pg_autoctl inspect command group provides read-only access to both local node state and cluster-wide state. These commands are safe to execute while pg_autoctl run is actively managing a node and do not require the PG_AUTOCTL_DEBUG environment variable to be set.

    Available sub-command groups include:

    • show: Network and hostname diagnostics.
    • pgsetup: Local PostgreSQL setup inspection.
    • fsm: Keeper Finite State Machine (FSM) state and transitions.
    • monitor: Querying the monitor's current state.
    • getpid: Retrieving PIDs for pg_autoctl sub-processes (services).

  6. Use pg_autoctl do for development and QA

    main

    The pg_autoctl do command group provides internal development and QA tooling, such as tmux session management and a demo application.

    Warning: This command group is not intended for production use.

    For production-related tasks, use the following instead:

    • Read-only diagnostics: Use pg_autoctl inspect.
    • Manual cluster recovery: Use pg_autoctl manual operations.

  7. Operate pg_auto_failover with pg_autoctl

    main

    The pg_autoctl utility is the primary tool for operating and running a pg_auto_failover installation. It supports two distinct modes of operation depending on the node type:

    1. Monitor mode: Used to run and manage the monitor node.
    2. Keeper mode: Used to run and manage keeper nodes (which include primary and standby Postgres instances).

    Most commands are compatible with both modes, allowing you to manage the lifecycle of the monitor and keeper nodes through a unified command-line interface.

  9. How Multiple Standby Architectures work

    main

    pg_auto_failover supports multiple standby nodes to increase data redundancy and availability. Depending on the configuration, you can achieve different trade-offs between performance and data safety.

    Replication Quorum and Async Standbys

    You can configure a subset of standbys to participate in the replication quorum while others remain asynchronous. For example, in a 4-node setup (1 Primary, 3 Standbys):

    • Quorum Nodes: Two standby nodes can participate in the quorum (e.g., setting number_sync_standbys = 1). This ensures at least two copies of the data exist (one on the primary and one on either of the quorum standbys).
    • Async Node: A third standby can be configured to not participate in the quorum. This is done by:
      • Ensuring the node is not in the synchronous_standby_names list.
      • Setting candidate-priority = 0 so the node is never a candidate for failover.

    This pattern is useful for cross-data center deployments where local nodes provide HA and a remote node provides Business Continuity or reporting capabilities.

  10. Understand the role of the pg_auto_failover Monitor

    main
    The pg_auto_failover monitor is a PostgreSQL extension responsible for managing the state machines of one or more server groups. Each group of servers represents a single Highly Available (HA) PostgreSQL Service. Multiple groups can be organized together into a single formation. The monitor acts as the central authority for coordinating failover and maintaining the desired state of the HA services.

  11. How pg_auto_failover handles fault tolerance and failover

    main

    pg_auto_failover uses a State Machine to manage high availability. The monitor drives the state machine by assigning goal states, while the keeper service on each node implements the transitions and reports success or failure back to the monitor.

    Key components of the lifecycle:

    • Monitor: Performs frequent health checks and orchestrates transitions by assigning new goal states.
    • Keeper: Responsible for implementing the transition to the assigned state and reporting status.
    • Retries: The keeper is permitted to retry transitions until they succeed or fail to reach the assigned state.

  12. How the Monitoring Protocol works

    main

    The Monitor and the Keeper (on data nodes) interact using two complementary protocols to ensure cluster health:

    1. Data Node Polling (Push/Pull): Data nodes periodically connect to the Monitor and execute SELECT pgautofailover.node_active(...). This communicates the node's current state and allows the Monitor to return the node's assigned goal state.
    2. Monitor Health Checks (Pull): The Monitor periodically connects to all data nodes to verify they are responsive, performing an operation equivalent to pg_isready.

    Health Logic: A node is only marked as unhealthy if the Monitor cannot connect to it and the node has not reported its state via node_active for a certain period. This prevents unnecessary failovers during routine PostgreSQL restarts.