Bosun Time Series Alerting Framework

Bosun is an open-source, MIT licensed, monitoring and alerting system developed by Stack Exchange. It features an expressive domain-specific language (DSL) for evaluating alerts and a template language for creating detailed notifications that include graphs, tables, and contextual information.

Key Capabilities:

• Time Series Evaluation: Use a flexible expression language to evaluate time series data.
• Historical Testing: Test alerts against historical data to reduce noise before production.
• Data Source Support: Queries OpenTSDB, Graphite, and Logstash-Elasticsearch.
• Multi-Platform: Runs on any operating system supported by Go (Linux, Windows, Mac).
• Dimension Support: Supports arbitrary dimensions and automatic incorporation of new tags (hosts, services, etc.).

scollector is a data collector designed as a replacement for OpenTSDB's tcollector. It is optimized for OpenTSDB 2.0 and integrates natively with Bosun.

Key advantages include:

• Uses the OpenTSDB v2 API (instead of the older v1 API).
• Higher resource efficiency compared to tcollector.
• Built-in support for various collectors across Windows, Linux, and Mac platforms.

Bosun's architecture consists of two primary components:

• scollector: A data collection agent. It is a binary that gathers local Linux and Windows data, polls network devices via SNMP and VSphere, and can run external scripts. It queues data if Bosun is unreachable and sends data to Bosun via compressed JSON to a REST API.
• bosun: The central engine for data collection/relaying, alerting, and graphing. It uses an expression language to create alerts from OpenTSDB time-series data and uses Go template language for notifications. It includes a web interface for dashboards, graphing, expression testing, and configuration validation.

Bosun provides a suite of Elastic functions to replace the deprecated ls (logstash) functions. These functions work with any Elastic documents containing a time field. They utilize two primary types:

1. ESIndexer: Generates index names to query based on date ranges.
2. ESQuery: Generates Elastic queries to filter results.

To view the generated JSON for queries during expression evaluation, use the miniprofiler by pressing Alt-P on the expr page.

Bosun's expression language uses four primary data types to process time series data into alert triggers:

• Scalar: A single numeric value with no associated group. Note that an empty group {} is still considered a group.
• NumberSet: A group of tagged numeric values, with one value per unique grouping. A Scalar can be used in place of a NumberSet if it has a single member with an empty group.
• SeriesSet: An array of timestamp-value pairs associated with a group. This is the typical format returned by time series databases.
• VariantSet: A generic type used by functions that can return a NumberSet, SeriesSet, or Scalar.

In most alerting workflows, you will receive SeriesSets from your database and use reduction functions to turn them into NumberSets.

Bosun provides a set of preview functions to query Azure Monitor for metric and resource information. These functions are available when AzureMonitorConf is defined in the system configuration.

Important Considerations:

• Rate Limits: Queries using az and azmulti are subject to Azure Resource Manager Request Limits. Bosun logs a warning when a request responds with fewer than 100 reads remaining. Excessive use of historical testing can quickly exhaust these limits.
• PrefixKey: You can use a quoted string as a PrefixKey before Azure query functions to query different Azure clients from a single Bosun instance. If no prefix is used, the query defaults to the standard Azure client.

Users can manage alerts through the following actions in the web interface:

• Acknowledge: Stops further notifications unless severity increases. Requires entering a name and a reason.
• Close: Removes the alert from the dashboard. Use this when an alert is handled. Note: Active alerts cannot be closed; they will reappear on the next schedule run.
• Forget: Used for active Unknown alerts (e.g., when a host is decommissioned). This is non-destructive; if data for that instance returns, the alert will reappear.
• History: Opens a timeline view of the selected alert instances.

Templates define the appearance of alerts when they are sent. Bosun uses Go's text/template for the subject (plaintext) and html/template for the body (HTML).

Key components of a template include:

• subject: The plaintext subject line, also used for dashboard incident titles.
• body: The HTML message body.
• Custom Fields: Any additional key/value pairs you add to a template can be used by notification systems as alternative content.

Note: Templates are rendered at the moment the expression is evaluated and becomes non-normal to ensure data consistency even if the underlying TSDB data changes later.

Notifications specify actions (like email or HTTP requests) to perform when alerts change severity or when users perform actions on incidents. Alerts reference notifications using the warnNotification and critNotification keywords. Notifications execute concurrently and can be chained together using next and timeout keywords to create escalation paths.

# HTTP Post to a chatroom, email in 10m if not ack'd
notification chat {
	next = email
	timeout = 10m
	post = http://chat.example.com/room/1?key=KEY&message=whatever
}

# email foo and bar each day until ack'd
notification email {
	email = foo@example.com, bar@example.com
	next = email
	timeout = 1d
}