Elasticsearch Curator Documentation

Curator uses filters to select specific indices or snapshots for actions. Filters are defined in YAML arrays using the filtertype key. You can chain multiple filters together by providing them as consecutive array elements in your configuration.

Index Filter Types: age, alias, allocated, closed, count, empty, forcemerged, kibana, none, opened, pattern, period, space.

Snapshot Filter Types: age, count, none, pattern, period, state.

- filtertype: age
  source: creation_date
  direction: older
  unit: days
  unit_count: 3
- filtertype: pattern
  kind: prefix
  value: log-

Elasticsearch Curator is a tool designed to manage Elasticsearch indices and snapshots through a three-step process:

1. Obtain an actionable list: It retrieves the full list of indices or snapshots from your Elasticsearch cluster.
2. Filter the list: It iterates through user-defined filters to progressively remove items from the actionable list based on your criteria.
3. Execute actions: It performs specified actions (such as deletion, rollover, or snapshotting) on the items that remain in the actionable list after filtering.

Curator provides a suite of tools to automate management tasks for both Elasticsearch indices and snapshots. Key capabilities include:

Index Management:

• Add or remove indices from an alias
• Change shard routing (allocation)
• Close or open indices
• Create new indices
• Delete indices
• Force merge indices
• Reindex indices (including from remote clusters)
• Change the number of replicas per shard
• Perform index rollover
• Shrink indices

Snapshot Management:

• Take snapshots (backups) of indices
• Delete snapshots
• Restore snapshots

Elasticsearch provides Index Lifecycle Management (ILM) as a built-in policy-based feature (available with at least a Basic license).

When to use ILM:

• If ILM provides the necessary functionality for your index lifecycle.
• If you want management to be handled as a matter of policy rather than execution-time analysis.
• If you are using Stack components (like Beats or Logstash) that use ILM by default.

When to use Curator:

• When you require execution-time analysis for management tasks.
• When you need to perform specific management actions that fall outside your defined ILM policies.

Index Lifecycle Management (ILM) applies specific policy actions to indices as they transition through time-oriented phases. Curator supports these ILM actions, which are triggered when indices enter the following phases:

• Hot
• Warm
• Cold
• Delete

Common ILM actions include:

• Set Priority
• Rollover
• Unfollow
• Allocate
• Read-Only
• Force Merge
• Shrink
• Delete

Elasticsearch Curator manages Elasticsearch indices and snapshots using a three-step workflow:

1. Obtain an actionable list: Curator retrieves a full list of indices or snapshots from your Elasticsearch cluster.
2. Filter the list: It iterates through user-defined filters to progressively remove items from the actionable list based on your criteria.
3. Execute actions: It performs specified actions on the remaining items in the actionable list.

To install a specific version of Curator, use the -U flag followed by == and the desired version number. The -U flag will uninstall any current version before installing the specified one.

pip install -U elasticsearch-curator==X.Y.Z

When using source: name with a timestring, Curator converts the strftime pattern into a regular expression. A pattern like %Y.%m (year and month) becomes ^.*\d{4}\.\d{2}.*$, which will match both monthly indices (index-2016.12) and daily indices (index-2017.04.01).

To prevent daily indices from being matched by a monthly pattern, use a second filter with exclude: True to filter out the more specific pattern.

- filtertype: pattern
  kind: timestring
  value: '%Y.%m'
- filtertype: pattern
  kind: timestring
  value: '%Y.%m.%d'
  exclude: True