Prometheus Alertmanager Documentation

The Alertmanager Mixin alerts are designed to monitor a cluster of Alertmanager instances. To ensure they function correctly, adhere to these requirements:

• Scraping: The Prometheus server evaluating the alerts must scrape all Alertmanager instances in the cluster, regardless of their physical location.
• Labeling: All Alertmanager instances within the same cluster must share the same job label.
• Isolation: If you are monitoring multiple distinct Alertmanager clusters, ensure that instances from different clusters use different job labels to prevent cross-cluster interference.

An Alertmanager cluster consists of multiple instances communicating via a gossip protocol (using Hashicorp's Memberlist).

Key architectural behaviors:

• Each instance receives alerts independently from Prometheus servers.
• Instances participate in a peer-to-peer gossip mesh.
• State (silences and notification logs) is replicated across members.
• Each instance processes and sends notifications independently.

This architecture ensures that if one instance fails, others in the cluster can continue to process alerts.

Alertmanager's High Availability (HA) implementation is built on three core principles to ensure reliable alert delivery:

1. Single pane view and management: Silences and alerts can be viewed and managed from any cluster member, providing a unified operational experience.
2. Survive cluster split-brain with "fail open": During network partitions, Alertmanager prefers to send duplicate notifications rather than miss critical alerts.
3. At-least-once delivery: The system guarantees that notifications are delivered at least once, prioritizing reliability over strict exactly-once semantics.

Use amtool config routes to view your configuration's routing tree as a text view. You can also test if a specific set of labels matches an expected receiver using the test subcommand.

# View routing tree of remote Alertmanager
$ amtool config routes --alertmanager.url=http://localhost:9093

# Test if alert matches expected receiver
$ amtool config routes test --config.file=doc/examples/simple.yml --tree --verify.receivers=team-X-pager service=database owner=team-X

To prevent all cluster members from sending duplicate notifications simultaneously, Alertmanager uses a staggered timing mechanism based on the peer's position in the sorted cluster list.

Calculation: wait_time = peer_position × peer_timeout

Example (3 instances, 15s peer timeout):

• Instance am-1 (position 0): waits 0 seconds
• Instance am-2 (position 1): waits 15 seconds
• Instance am-3 (position 2): waits 30 seconds

Position is determined by sorting all peer names alphabetically.

To install manual pages for amtool, generate the man page content using the --help-man flag and redirect it to your system's man directory (typically /usr/local/share/man/man1/amtool.1). You must run mandb to update the manual database so the new page is viewable.

amtool --help-man > /usr/local/share/man/man1/amtool.1
sudo mandb

Alertmanager HA is enabled by default. To form a cluster, instances must communicate via both UDP and TCP on the clustering port. Use the --cluster.* flags to configure peer communication.

Important Requirements:

• Both UDP and TCP must be whitelisted in firewalls and exposed in containers.
• The port in --cluster.listen-address must be specified in the --cluster.peer flag of other nodes.
• If an instance does not have an RFC 6890 IP address with a default route, --cluster.advertise-address is required.

To disable HA mode, set --cluster.listen-address= (an empty string).

To enable client authentication for HTTP traffic, set client_auth_type to RequireAndVerifyClientCert within the tls_server_config block. You must also provide a client_ca_file to validate the client certificates.

Additionally, you can use client_allowed_sans to restrict connections to clients whose certificate Subject Alternate Name (SAN) matches an entry in the provided list (supporting DNS, IP, e-mail, or URI).

tls_server_config:
  cert_file: <filename>
  key_file: <filename>
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: <filename>
  client_allowed_sans:
    - "example.com"
    - "192.168.1.1"

When using an Alertmanager cluster, do not use a load balancer between Prometheus and Alertmanager. A load balancer introduces a single point of failure and can interfere with the redundancy model.

Instead, configure Prometheus to send alerts to all Alertmanager instances directly using static_configs. This ensures that if one instance fails, others still receive the alerts and can process them independently.

# prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - am-1.example.com:9093
            - am-2.example.com:9093
            - am-3.example.com:9093

You can generate a bash completion script for the amtool CLI to improve shell usability. The generated script can be placed in /etc/bash_completion.d/ to enable autocompletion for your shell sessions.

amtool --completion-script-bash

You can install amtool, the CLI tool for interacting with the Alertmanager API, using Go:

$ go install github.com/prometheus/alertmanager/cmd/amtool@latest

If you update your configuration file, you can trigger a reload without restarting the process. If the new configuration is invalid, Alertmanager will log an error and continue using the previous valid configuration.

To reload, you can:

• Send a SIGHUP signal to the Alertmanager process.
• Send an HTTP POST request to the /-/reload endpoint.