DOCSEARCHbyAlgolia
Submit Docs

sparkMeasure

repository·master·Indexed 21 days ago

https://github.com/lucacanali/sparkmeasure

A performance troubleshooting tool and library for Apache Spark workloads. It provides a Python and Scala API to collect and analyze detailed stage, task, and memory metrics. Features include StageMetrics and TaskMetrics for different levels of granularity, as well as a Flight Recorder mode for streaming metrics to sinks like InfluxDB, Apache Kafka, and Prometheus Pushgateway. Supports Spark versions 2.1 through 4.x.

Tokens
34.2K
Snippets
116
Records
151
Agent score
73%

What's inside sparkmeasure

  1. Explore SparkMeasure usage modes

    master

    SparkMeasure can be used in several different ways depending on your environment and requirements:

    • Interactive mode: Best for spark-shell (Scala), PySpark (Python), or jupyter notebooks to collect and analyze metrics manually.
    • Batch and code instrumentation: Use the SparkMeasure API to instrument your code directly for collecting, saving, and analyzing metrics. This is ideal for performance testing pipelines.
    • Flight Recorder mode: Collects metrics transparently without code changes. Metrics can be saved to a file (local or Hadoop-compliant) or written in near-real-time to sinks like InfluxDB, Apache Kafka, Prometheus Pushgateway, or via JMX Exporter.

  2. Use Flight Recorder mode to instrument Spark applications

    master

    Flight Recorder mode allows you to instrument Spark applications without modifying their source code. It works by attaching a Spark Listener to the Spark Context, which collects execution metrics during runtime and saves them to a filesystem for later analysis.

    There are two levels of granularity:

    1. Stage-level granularity: Uses FlightRecorderStageMetrics. This is recommended for most applications as the data volume is relatively small ($O(\text{number of stages})$).
    2. Task-level granularity: Uses FlightRecorderTaskMetrics. This provides deeper detail but can generate very large amounts of data ($O(\text{number of tasks})$) in the driver. Use with caution.

  3. Compare InfluxDBSink and InfluxDBSinkExtended

    master

    SparkMeasure provides two primary sink classes for InfluxDB:

    1. InfluxDBSink: Collects and writes Spark metrics and application info in near real-time. The data volume is relatively small, scaling with the number of stages: $O(\text{number of stages})$.
    2. InfluxDBSinkExtended: Extends functionality to record metrics for every executed Task. Warning: This can generate a massive amount of data scaling with the number of tasks: $O(\text{number of tasks})$. Use with caution in large-scale applications.

    To enable stage-level metrics in the standard InfluxDBSink, you must set spark.sparkmeasure.influxdbStagemetrics to true.

  4. Understand key Spark metrics in sparkMeasure reports

    master

    When reviewing a sparkMeasure report, the following key metrics are provided (all time-based metrics are in milliseconds):

    • elapsedTime: Time taken by the stage or task to complete.
    • executorRunTime: Cumulative time spent by executors running the task.
    • executorCpuTime: Cumulative CPU time spent by executors running the task.
    • jvmGCTime: Time spent in garbage collection.
    • shuffle metrics: Details on I/O and time spent on shuffle operations.
    • I/O metrics: Details on reads and writes (Note: currently no time-based metrics for I/O operations).

  5. When to use Stage-level vs Task-level metrics

    master

    Choosing the right granularity depends on your analysis goals:

    • Use Stage-level metrics whenever possible. They are much more lightweight and provide aggregated performance metrics suitable for general workload monitoring.
    • Use Task-level metrics when you need deep granularity to investigate specific performance issues such as data skew, long tails, or task stragglers.

  6. How sparkMeasure works and its core concepts

    master

    sparkMeasure is a performance tool for Apache Spark built on the Spark Listener interface. It intercepts Spark executor Task Metrics and transports them from the executors to the driver.

    Key Abstractions

    • Granularity Modes: You can choose between Stage-level metrics (lightweight, aggregated) or Task-level metrics (detailed, useful for studying skew, long tails, and stragglers).
    • Reporting Modes:
      • Standard Mode: Metrics are buffered in the driver's memory (in a ListBuffer) and can be aggregated into reports, returned as Scala Maps/Python dictionaries, or converted into Spark DataFrames.
      • Flight Recorder Mode: Designed for high-volume data. Instead of buffering in memory, it writes metrics directly to external sinks to avoid driver memory bottlenecks.

    Supported Sinks

    • Standard Mode: Filesystems (via Hadoop API, including HDFS) or standard output.
    • Flight Recorder Mode: InfluxDB, Apache Kafka, or Prometheus Pushgateway.

  7. Compare KafkaSink and KafkaSinkV2

    master

    SparkMeasure provides several sink options for Kafka, depending on the level of detail required:

    KafkaSink Family

    • KafkaSink: Collects and writes Spark metrics to Kafka. Data volume is relatively small, proportional to the number of stages: $O(\text{number of stages})$.
    • KafkaSinkExtended: Extends KafkaSink to record metrics for every individual Task. Warning: This generates a large amount of data ($O(\text{number of tasks})$) and should be used with care.

    KafkaSinkV2 Family (Recommended)

    KafkaSinkV2 is an enhanced version that is backward compatible with the original KafkaSink but adds:

    • Application-Level Metrics: Emits applications_started and applications_ended events with aggregated counters.
    • Custom Labels: Support for metadata via spark.sparkmeasure.appLabels.*.
    • Enhanced End Events: Includes executor counts, job/stage/task counters, and selected Spark configurations.
    • Counter Tracking: Tracks success/failure counts for jobs, stages, and tasks.
    • KafkaSinkV2Extended: Extends KafkaSinkV2 to include detailed per-task metrics.

  8. How SparkMeasure Flight Recorder mode works

    master
    Flight recorder mode allows you to instrument Spark applications without modifying their source code. It works by attaching a SparkListener to the Spark application, which collects metrics and application information in near real-time while the application runs. This is particularly useful for monitoring Spark execution workloads and feeding metrics into dashboards like Grafana.

  10. Use PushGatewaySink in Flight Recorder mode

    master

    You can use sparkMeasure in Flight Recorder mode to instrument Spark applications without modifying their source code. This mode works by attaching a Spark Listener that collects metrics while the application runs.

    PushGatewaySink is a specific listener that collects Spark metrics (currently at the Stage level via StageMetrics) and writes them in near real-time to a Prometheus Pushgateway endpoint. This is useful for monitoring Spark execution workloads in environments where Prometheus needs to pull metrics from a gateway rather than directly from the Spark driver.

    # Example of attaching the listener via spark-submit or spark-shell
--conf spark.extraListeners=ch.cern.sparkmeasure.PushGatewaySink

  12. Add sparkMeasure to Spark runtime classpath

    master

    To make sparkMeasure available to your Spark session, use one of the following methods. The --packages method is preferred as it automatically resolves dependencies via Maven Central.

    Using --packages (Recommended)

    --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

    Using direct JAR/classpath methods

    --jars /path/to/spark-measure_2.13-0.28.jar
--jars https://github.com/LucaCanali/sparkMeasure/releases/download/v0.28/spark-measure_2.13-0.28.jar
--conf spark.driver.extraClassPath=/path/to/spark-measure_2.13-0.28.jar