ElastAlert Documentation

ElastAlert uses rule types to determine when a match is found based on Elasticsearch data. Common built-in rule types include:

• frequency: Match where there are at least X events in Y time.
• spike: Match when the rate of events increases or decreases.
• flatline: Match when there are less than X events in Y time.
• blacklist / whitelist: Match when a certain field matches a blacklist/whitelist.
• any: Match on any event matching a given filter.
• change: Match when a field has two different values within some time.
• new_term: Match when a never before seen term appears in a field.
• cardinality: Match when the number of unique values for a field is above or below a threshold.

When a rule match occurs, ElastAlert can trigger one or more alert types. Supported built-in alert types include:

• Email, JIRA, OpsGenie, Commands, HipChat, MS Teams, Slack, Telegram, GoogleChat, AWS SNS, VictorOps, PagerDuty, PagerTree, Exotel, Twilio, Gitter, Line Notify, Zabbix.

ElastAlert works by periodically querying Elasticsearch and processing the results through three main components:

1. Rule Types: Processes data returned from Elasticsearch queries. It determines if a match has occurred based on the rule's configuration and the data provided.
2. Alerts: Takes action when a rule type identifies a match. A match is typically a dictionary of values from an Elasticsearch document.
3. Enhancements: Intercepts the match dictionary before it is passed to the alert, allowing you to modify or enhance the data.

To use ElastAlert, you configure a set of rules where each rule defines a specific Elasticsearch query, a rule type, and one or more alerts.

Filters in ElastAlert rules are part of the Elasticsearch query DSL. The filter section in your rule configuration is passed to Elasticsearch to determine which results are passed to the rule for processing.

Note: For Elasticsearch 5.x and later, use query_string to implement boolean logic (AND, OR, NOT) instead of the nested and/or/not structures used in Elasticsearch 2.x.

If you deploy ElastAlert on an EC2 instance, you can assign an IAM role to the instance with permissions to read from and write to the Elasticsearch service.

To use an Instance Profile, you must provide the AWS region by either:

• Specifying aws_region in your ElastAlert configuration file.
• Setting the AWS_DEFAULT_REGION environment variable.

You can install the latest released version of ElastAlert using pip, or clone the repository for the most recent changes. If installing from source, you must install setuptools first.

Note: Depending on your Elasticsearch version, you may need to manually install the corresponding elasticsearch-py version.

• For Elasticsearch 5.0+: pip install "elasticsearch>=5.0.0"
• For Elasticsearch 2.X: `pip install "elasticsearch<3.0.0"

# Install via pip
$ pip install elastalert

# Or install from source
$ git clone https://github.com/Yelp/elastalert.git
$ pip install "setuptools>=11.3"
$ python setup.py install

ElastAlert uses an Elasticsearch index (defined by writeback_index in your global config) to store state, audit logs, and error information. This prevents data loss and duplicate alerts during restarts or crashes.

To create the index with the correct mappings, use the elastalert-create-index script. The script will prompt you for your cluster information (es_host, es_port). It also provides an option to copy documents from an existing ElastAlert writeback index.

When using Amazon Elasticsearch service, you must sign requests using AWS credentials to secure access. ElastAlert supports standard AWS credential methods:

1. Environment Variables: Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
2. AWS Config/Credential Files: Use files located at ~/.aws/config and ~/.aws/credentials.
3. AWS Instance Profiles: Uses the EC2 Metadata service (recommended for EC2 deployments).
4. AWS Profiles: Use specific named profiles via configuration or environment variables.

You can set any arbitrary JIRA field by prefixing the field name with jira_ in snake_case. You can use either the public-facing name or the internal representation (e.g., customfield_12345).

To use a value from the ElastAlert match in a custom JIRA field, prefix the field name with a # symbol.

jira_arbitrary_singular_field: My Name
jira_arbitrary_multivalue_field:
      - Name 1
      - Name 2
jira_customfield_12345: My Custom Value
jira_user: "#username"

To create a custom enhancement, subclass BaseEnhancement from elastalert.enhancements. Enhancements allow you to modify a match before an alert is sent. You must implement a process(self, match) method where match is a dictionary representing the alert data. You can modify the match dictionary in place to add or change fields.

from elastalert.enhancements import BaseEnhancement

class MyEnhancement(BaseEnhancement):
    def process(self, match):
        # Modify the match dictionary here
        if 'domain' in match:
            url = "http://who.is/whois/%s" % (match['domain'])
            match['domain_whois_link'] = url

For Elasticsearch 2.X

You can nest filters using not, and, and or keys.

For Elasticsearch 5.x+

Do not use nested boolean keys. Instead, use query_string to implement logic within a single query string.

# Elasticsearch 2.X style
filter:
- or:
    - term:
        field: "value"
    - wildcard:
        field: "foo*bar"
    - and:
        - not:
            term:
              field: "value"
        - not:
            term:
              _type: "something"

# Elasticsearch 5.x style
filter:
 - query:
      query_string:
        query: "somefield: somevalue OR foo: bar"