DOCSEARCHbyAlgolia
Submit Docs

Apache Spark Kubernetes Operator

repository·main·Indexed 43 days ago

https://github.com/apache/spark-kubernetes-operator

A Kubernetes operator for managing the lifecycle of Apache Spark applications and clusters using the Operator Pattern. It provides Custom Resource Definitions (CRDs) including SparkApp (v1) and SparkCluster (v1), supports Apache Spark 4.0+, and integrates with the Apache YuniKorn scheduler. Key features include Horizontal Pod Autoscaler (HPA) support for clusters, configurable resource retention policies, Prometheus metrics, and leader election for high availability.

Tokens
19.4K
Snippets
32
Records
49
Agent score
56%

What's inside Spark Kubernetes Operator

  1. Reset restart counters based on application stability

    main

    Use restartCounterResetMillis to allow long-running, stable applications to bypass restart limits. If an application attempt runs successfully for at least the specified duration (measured from DriverRequested), all restart counters (general and failure-specific) are reset to zero.

    This allows you to limit fast-failing restarts while permitting indefinite restarts for applications that demonstrate stability over time.

    restartConfig:
  ## 1hr
  restartCounterResetMillis: 3600000
  maxRestartAttempts: 3

  2. Deploy a Spark cluster using SparkCluster

    main

    The Spark Operator allows you to launch Spark clusters in Kubernetes using the SparkCluster custom resource. This resource allows you to specify desired master and worker instances with minimal effort.

    Implementation Details:

    • Workloads: Master and worker instances are deployed as Kubernetes StatefulSets.
    • Networking: Instances are exposed via Kubernetes services.
    • Configuration: You can specify the Spark version, worker count, and SparkConf.
    • Pod Templates: Similar to SparkApplication, you can submit pod templates for SparkCluster to configure specifications that are not supported via SparkConf. Note that Spark may overwrite certain fields in these templates.

  3. Suspend SparkApplication or SparkCluster resources

    main

    Both SparkApplication and SparkCluster custom resources support the .spec.suspend field. When set to true, the operator holds the resource in an initializing state (Submitted or ScheduledToRestart) and does not request the driver pod or the master/worker StatefulSets. Setting it back to false resumes the regular lifecycle.

    Key Behaviors:

    • Timing: suspend only takes effect before the driver (or master/worker) resources are requested. Setting it to true on a running application does not stop the current attempt. For a running cluster, setting it to true has no effect in the current version.
    • Restarts: If an application is configured to restart, setting suspend: true will cause the next attempt to be held until suspend is set back to false.
    • Status Visibility: For a new resource created with suspend: true, kubectl get will show an empty Current State because the Submitted status is not persisted to the API server until initialization resumes. An application held later in ScheduledToRestart will retain its previous status.
    • Deletion: Deleting a suspended resource works as usual.
    • Integration: This mechanism is designed to support external job queueing systems like Kueue by allowing them to admit workloads by flipping suspend to false.
    apiVersion: spark.apache.org/v1
kind: SparkApplication
metadata:
  name: suspended-pi
spec:
  suspend: true
  mainClass: "org.apache.spark.examples.SparkPi"
  jars: "local:///opt/spark/examples/jars/spark-examples.jar"
  runtimeVersions:
    sparkVersion: "4.2.0"

  4. How the Spark Kubernetes Operator works

    main

    The Spark-Kubernetes-Operator acts as a control plane for managing the lifecycle of Spark applications and clusters. It uses Custom Resources (CRs) to allow users to describe workloads via kubectl or the Kubernetes API.

    Core Workloads

    • SparkApplications: The operator launches a driver pod, observes its status, manages driver-spawned resources (like executors), and releases all owned resources once the application terminates.
    • SparkClusters: The operator launches a master and worker(s) based on the CR specification and releases resources upon failure.

    Implementation Details

    • Built with the Java Operator SDK for deployment and job submission.
    • Uses the fabric8 client to interact with the Kubernetes API Server.
    • The operator continuously tracks events through reconciliation loops.

  5. Understand SparkCluster state transitions

    main

    Unlike SparkApplication, which is expected to run and then terminate, a SparkCluster is intended to be a long-running resource.

    Lifecycle

    1. Submitted: The initial state.
    2. RunningHealthy: The cluster is active and running.
    3. Failed: The cluster encountered an error.
    4. ResourceReleased: The final state where the operator cleans up the cluster's Kubernetes resources.

    Similar to applications, Kubernetes resources created for a cluster are deleted by default at the end of its lifecycle.

  6. Manage Kubernetes Events for Spark Resources

    main

    When spark.kubernetes.operator.events.enabled is set to true, the operator publishes Kubernetes Event objects for SparkApplication and SparkCluster resources. These events are visible via kubectl describe or kubectl get events.

    Event Types and Reasons

    • Warning Events: Indicate issues such as SchedulingFailure, Failed, DriverEvicted, DriverStartTimedOut, DriverReadyTimedOut, ExecutorsStartTimedOut, RunningWithBelowThresholdExecutors, TerminatedWithoutReleaseResources (for SparkApplication), or CleanupError and StatusUpdateFailed (general operator errors).
    • Normal Events: Indicate successful state transitions like Submitted, DriverStarted, DriverReady, RunningHealthy, Succeeded, and ResourceReleased.

    Filtering Events

    To reduce noise, use spark.kubernetes.operator.events.excludedReasons to provide a comma-separated list of Java regular expressions. The operator will skip publishing events whose reason matches these expressions.

    Note: Use .* to exclude all reasons. Do not use {} or single * as they are invalid regex in this context.

    # Example: Exclude specific healthy states
spark.kubernetes.operator.events.excludedReasons=RunningHealthy,RunningWithPartialCapacity

# Example: Exclude all reasons using regex
spark.kubernetes.operator.events.excludedReasons=Running.*

  7. Use InstanceConfig to monitor Spark application health

    main

    instanceConfig helps the operator determine if an application is running healthily based on executor counts. This is useful in clusters without a batch scheduler to avoid hanging applications.

    • minExecutors: If the app has at least this many executors after starting, it is marked as RunningHealthy.
    • initExecutors: If the app acquires fewer than this many executors within .spec.applicationTolerations.applicationTimeoutConfig.executorStartTimeoutMillis, it is proactively shut down to avoid deadlock.
    • Threshold monitoring: If an app loses executors after a successful start, it is marked as RunningWithBelowThresholdExecutors.
    applicationTolerations:
  instanceConfig:
    minExecutors: 3
    initExecutors: 5
    maxExecutors: 10
sparkConf:
  spark.executor.instances: "10"

  8. Implement granular restart control for failures and scheduling

    main

    For fine-grained control, you can override default restart settings for specific failure types. The operator tracks three types of counters:

    1. General restart counter: Tracks all restarts.
    2. Consecutive failure counter: Tracks consecutive application failures (e.g., crashes). Resets to 0 on success.
    3. Consecutive scheduling failure counter: Tracks consecutive failures caused by the API server (e.g., quota exceeded, resource constraints). Resets to 0 on success.

    Limits are evaluated in this order:

    1. maxRestartAttempts (General limit)
    2. maxRestartOnSchedulingFailure (If applicable)
    3. maxRestartOnFailure (If applicable)

    If any limit is exceeded, the application stops.

    restartConfig:
  restartPolicy: Always
  # Default restart configuration (applies to all restarts)
  maxRestartAttempts: 5
  restartBackoffMillis: 30000  # 30 seconds

  # Override for consecutive general failures (application crashes, driver failures, etc.)
  # This counter resets to 0 on success
  maxRestartOnFailure: 3
  restartBackoffMillisForFailure: 60000  # 1 minute

  # Override for consecutive scheduling failures
  maxRestartOnSchedulingFailure: 1
  restartBackoffMillisForSchedulingFailure: 300000  # 5 minutes

  9. Configure Pod Templates for Driver and Executor

    main

    To configure Kubernetes pod specifications that are not available via standard SparkConf, you can use Pod Templates. The Operator supports two methods, with the first having higher precedence:

    1. In-spec PodTemplateSpec: Set .spec.driverSpec.podTemplateSpec or .spec.executorSpec.podTemplateSpec directly in the SparkApplication YAML. If this is used, the Operator will unset any spark.kubernetes.[driver/executor].podTemplateFile configurations to avoid conflicts.
    2. Remote File via SparkConf: Use the spark.kubernetes.[driver/executor].podTemplateFile configuration key.

    Note: If using a remote file (e.g., in S3 or Cloud Storage), the Spark Operator must have the necessary workload identity/permissions to access that location, and the driver pods will also need those permissions to create executors.

  10. Supported Custom Resource Definitions (CRDs)

    main

    The Apache Spark K8s Operator manages Spark workloads through the following Custom Resource Definitions (CRDs):

    • SparkApp: sparkapplications.spark.apache.org (version v1) - Used for managing Spark applications.
    • SparkCluster: sparkclusters.spark.apache.org (version v1) - Used for managing Spark clusters.

    Key Features

    • Support for Apache Spark 4.0+.
    • Support for Horizontal Pod Autoscaler (HPA) for SparkCluster resources.

  11. Deployment Topology and RBAC

    main

    A typical deployment involves placing the Operator pod in a dedicated namespace while it watches for SparkApplication and SparkCluster custom resources in one or more workload namespaces.

    Multi-instance Support

    Multiple Operator instances can coexist on the same cluster (e.g., per region or per tenant). To do this, each instance must use:

    1. A distinct ServiceAccount / RoleBinding name.
    2. A disjoint set of watched namespaces.

    RBAC Scoping

    RBAC is scoped via the Helm chart. The Operator's ServiceAccount is granted permissions to manage the specific Spark custom resources and their child Kubernetes objects.

  12. Configure Spark Application namespaces

    main

    By default, the operator creates Spark applications in the same namespace where the operator itself is deployed.

    If you need to run Spark applications in different namespaces, you must configure the Helm chart deployment to include the necessary RBAC resources for those additional namespaces.