Elastic Rally

Rally is a benchmarking tool designed for Elasticsearch. It automates several key tasks in the performance testing lifecycle:

• Cluster Management: Setup and teardown of Elasticsearch clusters specifically for benchmarking purposes.
• Data Management: Management of benchmark data and specifications across different Elasticsearch versions.
• Execution: Running benchmarks and recording the resulting data.
• Performance Analysis: Finding performance problems by attaching telemetry devices and comparing performance results.
• Reproducibility: Rally is designed to ensure that benchmarking data is reproducible.

A track is a specification for one or more benchmarking scenarios using a specific document corpus. It defines the environment and the workload for a benchmark.

Key attributes of a track include:

• Indices or data streams: The target data structures in Elasticsearch.
• Queries: The specific search operations to issue.
• Source URL: The location of the benchmark data.
• Challenges: A list of steps to run (e.g., indexing a specific number of documents or running searches for a set number of iterations).

A race is the execution of a benchmarking experiment. To run a race, you must select a track (the benchmarking scenario) and a challenge (the specific workload within that track).

Tracks are defined by their data and workload specifications. For more detailed information on specific tracks, you can refer to the rally-tracks repository. You can also create custom tracks to define your own benchmarking scenarios.

At the end of every Rally race, a summary report is generated. This report contains various performance and resource metrics. If you are building custom reports in Kibana, you can map the summary report lines to their corresponding metrics key values.

Note that not every summary report will display every line, and many time-based metrics (like indexing time) represent cumulative time across all threads (M threads * N minutes) rather than wall-clock time.

Certain names are now reserved as global variables in Rally Jinja2 track templates. If you have user-supplied track parameters with these names, you must rename them to avoid conflicts:

• serverless_operator: A boolean variable that is true if Rally targets a serverless cluster with operator privileges, and false otherwise (introduced in 2.10.0).
• build_flavor: A variable representing the build flavor reported by the Elasticsearch cluster (introduced in 2.9.0).

When running in a Serverless context, certain operations exhibit different default behaviors:

• Template Deletion: The delete-composable-templates operation ignores the delete-matching-indices property.
• Search Caching: The cache property in all search operations (search, paginated-search, scroll-search, composite-agg) defaults to false.

When using create-track, Rally generates a directory named after your track. The standard file structure includes:

• track.json: The core track definition containing the schedule and metadata.
• <index-name>.json: The mapping and settings for the extracted indices.
• <index-name>-documents.json(.bz2): The source documents for the index.
• <index-name>-documents-1k.json(.bz2): A smaller (1,000 document) version of the corpus used for --test-mode.
• operations/default.json: A sample operation for adding search requests.
• challenges/default.json: Initial actions for the default challenge (e.g., delete-index, create, bulk-index).

If you use an Elasticsearch metrics store, the relative-time metric is undergoing changes across versions. Follow this timeline to update your queries/dashboards:

• Rally 2.1.0: relative-time (microseconds) is deprecated; relative-time-ms (milliseconds) is added.
• Rally 2.2.0: relative-time is dropped. Only relative-time-ms is available.
• Rally 2.3.0: relative-time is reintroduced but will contain milliseconds. relative-time-ms becomes deprecated.
• Rally 2.4.0: relative-time-ms is dropped.

An optional meta structure can be defined at various levels to provide arbitrary key-value pairs. These properties are copied into each metrics record.

If the same key is defined at multiple levels, the most specific level overrides the more generic one. The hierarchy from generic to specific is:

1. track
2. challenge
3. operation
4. task

When a race completes, Rally outputs a summary report to the console. This report provides key performance metrics across different tasks. Common metrics include:

• Indexing/Merge/Refresh/Flush time: Cumulative and distribution (Min, Median, Max) times for various shard operations.
• Resource Usage: CPU usage, Garbage Collection (GC) time (Young Gen and Old Gen), and Heap usage (segments, doc values, terms, etc.).
• Storage: Store size, Translog size, and Index size.
• Throughput: Min, Mean, Median, and Max throughput (e.g., docs/s or ops/s) for specific tasks like index-append.
• Latency: Percentile latencies (50th, 90th, 99th, 99.9th, 100th) for both request latency and service time.
• Error Rate: The percentage of errors encountered during the task.

Rally supports different scheduling strategies to simulate client traffic:

• deterministic: Provides the highest level of reproducibility. It executes operations at specific, predictable intervals.
• poisson: Uses a Poisson distribution to model random, independent arrivals of clients. This is suitable for modeling real-world behavior where multiple clients decide independently when to issue a request (based on queueing theory).

You can also implement a custom schedule if your modeling needs are more complex.