DOCSEARCHbyAlgolia
Submit Docs

Cruise Control Documentation

repository·main·Indexed 25 days ago

https://github.com/linkedin/cruise-control

An operational tool for managing large-scale Apache Kafka clusters. Cruise Control automates workload balancing, resource utilization tracking, and cluster self-healing to address broker failures and resource capacity violations. It features a pluggable goal system for cluster optimization, anomaly detection, and workload modeling for CPU, disk, and network metrics. Compatible with various Apache Kafka versions depending on the branch, requiring Java 17.

Tokens
40.8K
Snippets
45
Records
165
Agent score
83%

What's inside Cruise Control

  1. Overview of Cruise Control features

    main

    Cruise Control is designed for hands-free Kafka cluster operations. Key out-of-the-box features include:

    • Cluster Balancing: Continually balances Kafka clusters based on disk, network, and CPU utilization.
    • Automatic Replica Reassignment: Automatically reassigns replicas from failed brokers to other brokers to restore the original replication factor.
    • Resource Identification: Identifies topic-partitions that consume the most resources in the cluster.
    • Cluster Lifecycle Management: Supports one-click cluster expansions and broker decommissions.
    • Heterogeneous Support: Supports heterogeneous Kafka clusters and multiple brokers per machine.

  2. Understand the Cruise Control workload model resources

    main

    Cruise Control builds its cluster workload model using four primary resource metrics. To achieve the necessary replica-level granularity for cluster modeling, Cruise Control derives CPU and network metrics from these raw inputs:

    | Resource | Granularity of Raw Metric | | :--- | :| | CPU | Per broker | | Disk Utilization | Per replica | | Bytes In Rate | All leader replicas of a topic per broker | | Bytes Out Rate | All leader replicas of a topic per broker |

  3. Understand the Load Monitor and Cluster Load Model

    main

    The Load Monitor is responsible for collecting Kafka metrics and generating a Cluster Load Model.

    Key Functions

    • Metric Collection: Collects standard Kafka metrics and derives per-partition resource metrics (e.g., estimated CPU utilization).
    • Cluster Load Model: Produces a software model reflecting the current replica assignment with replica-granularity load data (disk, CPU, bytes-in, and bytes-out).
    • Simulation: The model provides interfaces to simulate the impact of moving partitions or replicas, which the Analyzer uses to generate solutions.

    Components

    • Metric Fetcher Manager: Coordinates sampling tasks using a configurable number of threads.
    • Sampling Tasks: Includes Metric Sampling Task, Bootstrap Task, and Linear Model Training Task.
    • Metric Sample Aggregator: Organizes samples into 'load windows' (e.g., 1-hour windows). You can define a minimum number of samples required for a window to be considered valid.
    • Sample Store: A pluggable component that can save metric and training samples to external storage.

  4. Monitor Cruise Control status using Dropwizard metrics

    main
    Cruise Control uses Dropwizard metrics to report its internal status. These metrics are exposed via MBeans and can be used to monitor the health and performance of various Cruise Control components, including the Executor, LoadMonitor, UserTaskManager, AnomalyDetector, GoalOptimizer, MetricFetcherManager, and KafkaCruiseControlServlet.

  5. Execute Optimization Proposals with the Executor

    main

    The Executor carries out the optimization proposals generated by the Analyzer.

    Key characteristics:

    • Interruptible: Designed to be safely interrupted during execution.
    • Resource-Aware: Ensures that the execution process does not overwhelm any Kafka broker.

  6. Cruise Control Anomaly Detection and Self-Healing

    main

    Cruise Control monitors the cluster for anomalies and can be configured to take automated actions (self-healing).

    Detected Anomalies:

    • Broker failure
    • Goal violation
    • Metric anomaly
    • Disk failure
    • Slow brokers
    • Topic replication factor anomaly
    • Topic partition size anomaly
    • Maintenance Events

    Self-Healing Actions:

    • fix: Resolve the problem immediately (e.g., start a rebalance or fix offline replicas).
    • check: Re-evaluate the situation after a configurable grace period.
    • ignore: Disable self-healing for the specific anomaly.

  7. Configure Optimization Goals in the Analyzer

    main

    The Analyzer generates optimization proposals based on user-provided goals. Goals are categorized into two types:

    1. Hard Goals: Must be satisfied (e.g., rack-aware replica placement). If an optimization violates a hard goal, the process fails.
    2. Soft Goals: May be left unmet if necessary to satisfy all hard goals.

    Supported Goals

    • Replica Placement: Must be rack-aware.
    • Resource Utilization: Broker resource utilization and disk utilization must stay within pre-defined thresholds.
    • Network Capacity: Network utilization must not exceed capacity, even if all replicas on a broker become leaders.
    • Uniformity:
      • Uniform resource utilization across all brokers.
      • Uniform bytes-in rate of leader partitions across brokers.
      • Even distribution of partitions of a specific topic across brokers.
      • Even distribution of replicas (globally) across brokers.
      • Even distribution of leader replicas (globally) across brokers.
      • Even distribution of disk utilization across disks of each broker.

    Users can implement and plug in their own custom goals.

  10. Best practices for implementing Cruise Control goals

    main

    When implementing a new Goal, follow these best practices to ensure cluster stability and compatibility with existing optimizations:

    • Check Cluster Healthiness: Before proposing optimizations, verify if the cluster has dead brokers. Unhealthy clusters often require manual intervention or specific recovery steps rather than standard balancing.
    • Respect Previously Optimized Goals: Ensure your proposed operations do not violate requirements set by higher-priority goals. You can do this by invoking the isProposalAcceptable(BalancingProposal, ClusterModel) method of the previously optimized goals.
    • Handle Excluded Topics: Always check the excluded topics list to ensure that topics marked for exclusion are not modified by your goal.

  11. Configure HTTP Basic Authentication

    main

    Cruise Control provides simple HTTP Basic authentication by default using the com.linkedin.kafka.cruisecontrol.servlet.security.BasicSecurityProvider class.

    To enable it, set webserver.security.enable=true.

    Credentials must be stored in a file specified by webserver.auth.credentials.file. The file must follow the Jetty HashLoginService format:

    username: password [,rolename ...]

    Ensure this file is stored in a secure, protected location accessible only by Cruise Control.

  12. Resolve environment variables in configuration files

    main

    To avoid hardcoding sensitive information like passwords in configuration files, you can resolve environment variables directly within Cruise Control configs using the ${env:VARIABLE_NAME} syntax. This allows the system to inject the value of the specified environment variable at runtime.

    webserver.ssl.keystore.password=${env:SSL_KEYSTORE_PASSWORD}