PrometheusAlert Documentation

PrometheusAlert is an open-source operations and maintenance (O&M) alert center message forwarding system. It acts as a middleware that receives alert messages from various monitoring and logging systems via WebHooks and forwards them to multiple notification channels.

Supported Sources (via WebHook):

• Prometheus, Zabbix, Graylog2, Graylog3, Grafana, SonarQube, Alibaba Cloud CloudMonitor, and any system supporting WebHooks.

Supported Notification Targets:

• DingTalk, WeChat, Email, Feishu, Tencent SMS/Voice, Alibaba Cloud SMS/Voice, Huawei SMS, Baidu Cloud SMS, Ronglian Cloud Voice, Qimo SMS/Voice, Telegram, Baidu Hi (RuLiu), Kafka, etc.

PrometheusAlert provides a custom template interface at /prometheusalert that accepts HTTP POST requests containing JSON data (typically from systems with WebHook support).

Workflow:

1. An external system (e.g., Prometheus) sends a POST-JSON request to /prometheusalert.
2. The request URL includes a tpl parameter specifying which template to use and a type parameter (e.g., dd for DingTalk).
3. PrometheusAlert uses the specified template to render the JSON data into a text format.
4. The rendered text is forwarded to the target destination (e.g., a DingTalk robot URL provided in the request).

PrometheusAlert works by providing a custom template interface at /prometheusalert. This endpoint receives messages from various alerting systems or any system with WebHook capabilities. The received messages are then rendered using custom templates and forwarded to different target receivers.

Integration Workflow:

1. Install PrometheusAlert: Follow the installation guide.
2. Configure app.conf (Optional): If you are only using DingTalk, WeChat Work, or Feishu robots, you may skip this. However, if you need to use SMS, phone calls, or email, you must configure the relevant items in app.conf first.
3. Configure the Alerting System: Set up your specific alerting system (e.g., Prometheus, Grafana, Zabbix) to send data to the /prometheusalert endpoint.

PrometheusAlert for Zabbix uses a Client/Server architecture. You must deploy the zabbixclient on the same server as your zabbix-server.

To install, download the client to the Zabbix alertscripts directory and set the appropriate permissions.

# Download the client to the Zabbix alertscripts directory
wget https://raw.githubusercontent.com/feiyu563/PrometheusAlert/master/zabbixclient/zabbixclient -O /usr/lib/zabbix/alertscripts/zabbixclient

# Set ownership and permissions
chown zabbix:zabbix /usr/lib/zabbix/alertscripts/zabbixclient
chmod 755 /usr/lib/zabbix/alertscripts/zabbixclient

To enable the hot reload feature, you must modify the conf/app.conf configuration file before starting the application. Set the open-hotreload key to 1.

Important: If you do not enable open-hotreload before the initial program startup, enabling it via the API later will not work because the in-memory configuration will still have the feature disabled. You must restart the program after changing this setting in the config file.

# conf/app.conf
# Enable hot reloading
open-hotreload=1

To enable WeChat Work (WeCom) alerts, you must first create a robot in your WeChat Work group.

1. Open WeChat Work and enter the target group.
2. Go to Group Settings -> Group Robot -> Add.
3. Copy the generated Webhook URL.

In the app.conf configuration file, enable the WeChat channel and provide the URL using the following keys:

• open-weixin: Set to 1 to enable, 0 to disable.
• wxurl: The Webhook URL copied from WeChat Work.

Mentioning Users: You can mention specific users using the @username syntax. To do this, you must obtain the user's account ID from the WeChat Work management backend.

# 是否开启微信告警通道,可同时开始多个通道0为关闭,1为开启
open-weixin=1
# 默认企业微信机器人地址
wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxx

For the custom template interface /prometheusalert, you can pass the alertgroup parameter in the URL query string. This allows you to use short group names instead of long, complex URLs containing multiple webhook addresses.

Address Resolution Priority:

1. Alert Group (defined in app.conf)
2. URL Parameters
3. Default Configuration

Note: Parameters like type, tpl, webhookContentType, at, rr, and split should still be passed as URL parameters and are not handled within the alert group configuration sections.

PrometheusAlert supports alert routing for Prometheus alert messages. Routing works by filtering labels from the incoming Prometheus alert messages to forward them to specific templates and corresponding receiving targets. This design is similar to Prometheus Alertmanager's routing functionality.

To implement routing, you define rules that match specific labels in the alert. When a match is found, the alert is rendered using a specified template and sent to the configured destination (e.g., DingTalk, Feishu, or phone calls).

If your DingTalk robot requires a signature (加签), follow these steps:

1. In DingTalk, enable the 'Signature' (加签) option when creating the robot and copy the secret key.
2. In your app.conf, set open-dingding-secret=1.
3. Append the secret to your ddurl using the secret parameter: https://oapi.dingtalk.com/robot/send?access_token=XXX&secret=mysecret.

PrometheusAlert will automatically process this secret to generate the final signed URL containing timestamp and sign parameters.

Important: If open-dingding-secret is enabled in the config but the ddurl does not contain a secret parameter, the program will fallback to using the unsigned URL.