Pacemaker Documentation

Pacemaker is a high-availability cluster resource manager designed to preserve service integrity and minimize downtime by running on a set of hosts (a cluster).

Key features include:

• Failure Detection & Recovery: Detects and recovers from node- and service-level failures.
• Fencing (STONITH): Ensures data integrity by isolating faulty nodes using fence devices (e.g., power or network switches).
• Flexible Redundancy: Supports various configurations like Active/Passive, N+1, and N-to-N.
• Resource Management: Supports multiple interface standards, including clones (running on multiple nodes), promotable clones (running in specific roles), and containerized services.
• Cluster-wide Relationships: Allows specifying ordering, colocation, and anti-colocation between services.
• Replicated Configuration: Configuration is automatically replicated and can be updated from any node in the cluster.

The Pacemaker Cluster Test Suite (CTS) provides testing tools for installed environments. It is divided into two main categories:

1. Regression Tests: These test individual Pacemaker components in isolation. The primary entry point is cts-regression, which acts as a wrapper for component-specific tests: cts-cli, cts-exec, cts-fencing, and cts-scheduler.

   • Note: cts-cli and cts-scheduler can run from a source distribution.
   • Note: Other regression tests require an installed environment and must be run on a node where a cluster is not currently running.

2. The CTS Lab: A cluster exerciser designed to intensively test the behavior of an entire working cluster by running a randomized series of predefined tests. It can either run against an existing cluster configuration or overwrite it with a test configuration. Most lab logic resides in the Pacemaker Python module, with cts-lab serving as the front end.

Resource templates can be used directly in constraints and resource sets to apply logic to all primitives derived from that template, serving as an alternative to resource sets.

Supported Constraints

• order constraints
• colocation constraints
• rsc_ticket constraints

Usage Rules

• Colocation: In a colocation constraint, you can reference a template in either rsc or with-rsc, but the other side must be a regular resource (not another template).
• Resource Sets: Templates can be used within <resource_set> inside order or colocation constraints. If the resources in the set can run in parallel, use sequential="false" on the resource set containing the template reference.

<!-- Colocate all VMs from a template with a base resource -->
<rsc_colocation id="vm-template-colo-base-rsc" rsc="vm-template" rsc-role="Started" with-rsc="base-rsc" score="INFINITY"/>

<!-- Order: base-rsc -> (all vm-template resources in parallel) -> top-rsc -->
<rsc_order id="order2" score="INFINITY">
  <resource_set id="order2-0">
    <resource_ref id="base-rsc"/>
  </resource_set>
  <resource_set id="order2-1" sequential="false">
    <resource_ref id="vm-template"/>
  </resource_set>
  <resource_set id="order2-2">
    <resource_ref id="top-rsc"/>
  </resource_set>
</rsc_order>

The join sequence (or join process) is the mechanism by which nodes become ready to be assigned tasks in the cluster. While most daemons use Corosync membership, the controller requires nodes to explicitly join through several phases.

A node's current phase is tracked in the user_data member of pcmk__node_status_t using the enum controld_join_phase.

The Join Lifecycle:

1. None: Initial state (controld_join_none).
2. Welcomed: The DC sends a join offer (CRM_OP_JOIN_OFFER), and the node moves to controld_join_welcomed.
3. Integrated: The node responds with a join request (CRM_OP_JOIN_REQUEST). Once the DC receives this, the node moves to controld_join_integrated.
4. Finalized: The DC syncs the current best CIB to the integrated node. Upon successful sync, the DC sends an acknowledgment (CRM_OP_JOIN_ACKNAK), and the node moves to controld_join_finalized.
5. Confirmed: The node confirms the finalization (CRM_OP_JOIN_CONFIRM). The node then moves to controld_join_confirmed once the DC acknowledges (CRM_OP_JOIN_ACKNAK).

Once all nodes reach the confirmed state, the DC performs a final check for quorum via do_dc_join_final().

While Pacemaker does not strictly rely on synchronized time or time zones, synchronization is highly recommended for:

• Date/time-based rules: Ensuring rules evaluate consistently across all nodes regardless of which node is the Designated Controller (DC).
• Log Analysis: Facilitating easier comparison of logs across multiple nodes during troubleshooting.

Clock Jump Risks

• Jumping Forward: May cause various timeouts to expire prematurely.
• Jumping Backward: Can cause serious cluster problems and should be avoided.

Best Practices

If the host clock is adjusted at boot, Pacemaker's start should be ordered after the clock adjustment.

• systemd: Pacemaker automatically orders itself after time-sync.target.
• Workarounds: Depending on your setup, you may need to enable services like chronyd-wait.service or implement custom workarounds to ensure time is synchronized before Pacemaker starts.

You can control how Pacemaker reacts to repeated resource failures using resource meta-attributes.

Migration Threshold

Set the migration-threshold meta-attribute to define how many failures are allowed before a resource is banned from its current node and moved elsewhere.

• The threshold is applied to the resource as a whole (the sum of all operation fail counts).
• If migration-threshold is N, the resource moves after N failures.

Automatic Failure Expiration

Set the failure-timeout meta-attribute to allow fail counts to expire automatically.

• failure-timeout is measured from the most recent failure.
• If no new failure occurs within the timeout period, all existing failures are cleared (reset to 0) simultaneously.

Critical Failure Behaviors

• Start Failures: If the cluster property start-failure-is-fatal is true (default), a failed start sets the fail count to INFINITY, causing an immediate move.
• Stop Failures: If a resource fails to stop and fencing is enabled, the cluster will fence the node to allow the resource to start elsewhere. If fencing is disabled, the cluster will attempt to stop it again after the failure-timeout or manual cleanup.

# Example configuration logic:
# migration-threshold=2, failure-timeout=60s
# Result: Resource moves after 2 failures; can move back after 60s of stability.