DOCSEARCHbyAlgolia
Submit Docs

Kubeflow Spark Operator

repository·master·Indexed 25 days ago

https://github.com/kubeflow/spark-operator

A Kubernetes Operator for Apache Spark that enables the management of Spark applications using Kubernetes Custom Resources. It automates spark-submit and provides features such as cron scheduling via ScheduledSparkApplication and automatic retries. The project includes a Python API (kubeflow_spark_api) for programmatic management, Helm charts for deployment, and support for Prometheus metric exporting.

Tokens
33.1K
Snippets
77
Records
175
Agent score
83%

What's inside kubeflow-spark-operator

  1. Overview of Kubeflow Spark Operator

    master

    The Kubeflow Spark Operator is a Kubernetes Operator for Apache Spark that allows you to run Spark applications as declarative Kubernetes workloads. Instead of manually running spark-submit, you define your job in a YAML SparkApplication custom resource and apply it using kubectl.

    Key capabilities include:

    • Declarative Applications: Manage Spark jobs via the Kubernetes API using the SparkApplication resource.
    • Native Cron Scheduling: Schedule jobs using ScheduledSparkApplication with configurable concurrency policies and history limits.
    • Pod Customization: Use a mutating admission webhook to mount ConfigMaps/volumes and set affinity or tolerations for driver and executor pods.
    • Metrics & Monitoring: Export application and pod-level metrics to Prometheus, with optional JMX exporter support.
    • Batch Scheduling: Integration with Volcano, Apache YuniKorn, and Kubernetes scheduler plugins for gang scheduling.
    • Production Readiness: Automatic application restarts, retries with backoff, and resubmission of updated specs.

  2. Understand Spark Operator performance bottlenecks

    master

    When benchmarking the Kubeflow Spark Operator, focus on the operator's ability to manage high job submission rates and pod launches rather than the Spark jobs themselves. To isolate the operator's performance, use minimal resources for driver and executor pods.

    Performance is influenced by:

    • Controller Processing: The efficiency of the reconciliation loop processing SparkApplication CRDs.
    • Resource Allocation: The CPU and memory allocated to the controller pod (especially for JVM processes spawned per submission).
    • Work Queue: The single work queue per operator instance managed via controller-runtime.
    • Kubernetes API: The responsiveness of the Kubernetes API server during job translation and status updates.
    • Webhooks: The latency introduced by validation webhooks (disabling them can save ~60s per job).

  5. Understand Spark Operator Architecture

    master

    The operator consists of four primary components that work together to manage Spark workloads:

    1. SparkApplication Controller: Watches for SparkApplication custom resources (creation, updates, deletion) and enqueues them for processing.
    2. Submission Runner: Receives submission requests from the controller and executes the spark-submit command to launch the application.
    3. Spark Pod Monitor: Watches Spark pods and reports their status back to the controller to update the SparkApplicationStatus.
    4. Mutating Admission Webhook: Handles pod customizations (like mounting ConfigMaps or volumes) by intercepting pod creation requests and applying configurations based on annotations added by the controller.

    Workflow:

    1. User creates a SparkApplication object via kubectl.
    2. Controller creates a submission and sends it to the Submission Runner.
    3. Submission Runner starts the driver pod, which then creates executor pods.
    4. Pod Monitor tracks pod status and updates the SparkApplication status field.

  6. Test and document Helm charts

    master

    For developers working on the Helm charts:

    • Linting: Run make helm-lint to validate the charts.
    • Unit Testing: Run make helm-unittest to execute Helm chart unit tests (requires helm-unittest).
    • Documentation: The Helm chart README.md is generated from README.md.gotmpl using helm-docs. To update it, run make helm-docs.

    Note: If git pre-commit hooks are installed, helm-docs runs automatically during commit.

    make helm-lint
make helm-unittest
make helm-docs

  8. Enable or disable Prometheus metrics for the Spark Operator

    master

    The Spark Operator exposes metrics for Prometheus scraping. By default, the Helm chart installs the operator with metrics enabled (-enable-metrics=true).

    To install the operator without metrics enabled, use the following Helm command:

    helm install my-release spark-operator/spark-operator \
    --namespace spark-operator \
    --create-namespace \
    --set metrics.enable=false

    If you manually change the port or endpoint, ensure you also update the prometheus.io/port, prometheus.io/path, and containerPort annotations in your deployment manifest.

  9. Run Spark Operator Benchmarks using Locust

    master

    The benchmarking test kit uses Locust to simulate concurrent SparkApplication submissions. The script generates unique job names using uuid.uuid4(), respects rate limits, and monitors job status.

    Installation

    Install Locust and its dependencies in a virtual environment:

    python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

    Locust CLI Arguments

    Use the following flags to configure the benchmark:

    • -u, --users: Number of concurrent virtual users.
    • --jobs-per-min: Rate of job submissions.
    • --job-limit-per-user: Maximum number of jobs to submit for each user.
    • --spark-namespaces: Comma-separated list of namespaces where jobs will be submitted.
    • --spark_job_template: Path to the SparkApplication YAML template.
    • --no-delete-jobs: If set, jobs are not deleted after the test (useful for debugging).

    Example: Headless Benchmark Run

    To run a headless benchmark submitting 6000 jobs (2000 per user) at a rate of 1000 jobs per minute across three namespaces:

    locust --headless --only-summary -u 3 -r 1 \
--job-limit 2000 \
--jobs-per-min 1000 \
--spark-namespaces spark-team-a,spark-team-b,spark-team-c

  11. Integrate Kubeflow Notebooks with Spark Operator via Jupyter Enterprise Gateway

    master

    You can run PySpark workloads at scale directly from a Kubeflow notebook interface by integrating the Spark Operator with Jupyter Enterprise Gateway. This allows notebooks to execute PySpark code remotely using Kubernetes resources managed by the Spark Operator.

    Prerequisites

    • A running Kubeflow deployment with Notebook Controller enabled.
    • Spark Operator installed and configured in the cluster.
    • Helm installed locally.