Reaper for Apache Cassandra

Reaper is an open-source tool designed to schedule and orchestrate repairs for Apache Cassandra clusters. It enhances the standard nodetool repair process by providing:

• Tunable Segmentation: Splitting repair jobs into smaller, manageable segments.
• Back-pressure Management: Monitoring active repairs and pending compactions to prevent cluster overload.
• Control and Visibility: The ability to pause, cancel, and precisely track the progress of repairs.

Reaper can be interacted with via a REST API, a command line tool, and a web UI.

Reaper is a centralized, stateful, and highly configurable tool designed for running Apache Cassandra repairs, specifically for multi-site clusters. It replaces manual nodetool commands and crontabs with automated, intelligent coordination.

Key Capabilities

• Centralized Management: Manage repairs across all Cassandra clusters from a single location.
• Intelligent Scheduling: Automatically schedules repairs to avoid excessive cluster load and prevents overlapping repairs.
• Fault Tolerance: Includes built-in retry logic and state management.
• Granular Control: Schedule repairs at the keyspace, table, or token range level.
• Multi-Datacenter Awareness: Coordinates repairs across multiple sites and datacenters.
• Extensibility: Provides a full REST API for programmatic control and supports multiple backends for storing Reaper's own state data.

Reaper is a centralized, stateful, and highly configurable tool designed to run Apache Cassandra repairs against single or multi-site clusters.

Key features include:

• Segmented Repairs: Supports running repairs in a segmented manner.
• Parallelism: Opportunistically runs multiple parallel repairs simultaneously on different nodes within a cluster.
• Scheduling: Provides basic repair scheduling functionality.

The Reaper UI is built using the following technologies:

Libraries:

• React
• RxJS
• Bootstrap (CSS)

Tooling:

• webpack
• Babel (ES6)
• react-hot-reloader

Reaper uses Role-Based Access Control (RBAC) with two distinct roles:

• user: Provides read-only access. Users can view clusters, repair runs, schedules, and metrics, but cannot create, modify, or delete any resources. Ideal for monitoring and reporting.
• operator: Provides full access. Users can create, modify, and delete clusters, repair runs, and schedules, and can trigger repairs. Ideal for administrators.

The Reaper system is composed of several key architectural elements:

• State Database: Contains the full state of the system. Reaper does not use internal caches for state changes (such as running repairs or registered clusters), ensuring that changes made directly to the storage reflect dynamically in the running system.
• RESTful API: Provides the programmatic interface for the system.
• spreaper CLI: A command-line tool that serves as an alternative way to issue commands to a running Reaper instance.
• JMX Communication: Reaper communicates with Cassandra nodes in registered clusters via JMX.
• Dropwizard: The project is built on top of the Dropwizard framework.

Note on Storage: While Reaper typically uses a persistent database, it can be run with memory storage for testing purposes. However, memory storage is not persistent.

Reaper divides repair runs into Segments. A segment is a subrange of tokens that fits within a single cluster token range.

• Minimum Segments: The number of segments is at least equal to the number of token ranges in the cluster (e.g., a 3-node cluster with 256 vnodes per node will have at least 768 segments).
• Optimization: Since Reaper 1.2.0 and Cassandra 2.2, token ranges sharing the same replicas can be consolidated into a single segment to reduce overhead.
• Cassandra 3.0+: Cassandra generates a single repair session for all subranges sharing the same replicas, further reducing vnode overhead.

Address translator mapping allows you to map hostnames to specific IP addresses, which is useful in environments like EC2 multi-region or multi-IP per node setups. You can configure this for both the Reaper connection and the JMX connection using the following variables:

• REAPER_CASS_ADDRESS_TRANSLATOR_TYPE / JMX_ADDRESS_TRANSLATOR_TYPE: The type of translator (e.g., ec2MultiRegion or multiIpPerNode).
• REAPER_CASS_ADDRESS_TRANSLATOR_MAPPING / JMX_ADDRESS_TRANSLATOR_MAPPING: The mapping in host:ip format.

REAPER_CASS_ADDRESS_TRANSLATOR_TYPE=multiIpPerNode
REAPER_CASS_ADDRESS_TRANSLATOR_MAPPING=host1:ip1,host2:ip2

This results in the following configuration in cassandra-reaper.yaml:

addressTranslator:
    type: multiIpPerNode
    ipTranslations:
    - from: "host1"
      to: "ip1"
    - from: "host2"
      to: "ip2"

Reaper uses a backend to store its own metadata and state. The backend has evolved to prioritize simplicity and low resource consumption:

• Current Backend (v4.2.0+): Uses SQLite. This was chosen over previous options to reduce memory consumption during long-term operation.
• Legacy Backends (Removed):
  • Postgres and H2 (Removed in v3.0.0).
  • EclipseStore (Used between v3.6.0 and v4.1.x; replaced by SQLite in v4.2.0).

Reaper is compatible with a wide range of Apache Cassandra versions:

• General Support: All versions from 1.2 up to the latest release.
• Incremental Repair: Supported for Cassandra versions 2.1 and above.

A single Reaper instance is capable of managing repairs for multiple clusters even if they are running different versions of Apache Cassandra.

While nodetool repair can be scheduled via crontab, Reaper provides several advantages for managing Cassandra repairs:

• Intelligent Scheduling: Reaper automatically schedules repairs to minimize cluster load and avoid performance impact, whereas manual cron jobs require complex staggering to prevent overlaps.
• Granular Control: Reaper provides a UI to schedule repairs with high granularity.
• Cluster Awareness: Reaper manages the repair process across the cluster more effectively than isolated cron jobs.