DOCSEARCHbyAlgolia
Submit Docs

SparkPlusPlus

website·Indexed 29 days ago

https://sparkplusplus.github.io

A Scala framework for building Apache Spark applications with reduced boilerplate. It provides a structured approach to Spark apps using SparkApp and SparkETLApp entrypoints, typed YAML configuration, and a utility layer for common DataFrame tasks. The framework supports Scala 2.12 and 2.13 and is designed to standardize config loading, SparkSession creation, and logging for batch jobs.

Tokens
5.9K
Snippets
21
Records
40
Agent score
72%

What's inside SparkPlusPlus

  1. Overview of SparkPlusPlus for Apache Spark

    SparkPlusPlus is a Scala framework designed for building Apache Spark batch applications. It reduces boilerplate and provides a consistent runtime model by offering standardized mechanisms for:

    • Loading typed YAML configurations.
    • Creating and customizing the SparkSession.
    • Declaring input and output datasets via YAML.
    • Passing runtime context into application logic.
    • Organizing reusable DataFrame utilities alongside Spark jobs.

  2. Overview of SparkPlusPlus Core Components

    SparkPlusPlus provides a lightweight framework on top of Apache Spark to avoid the need for a full orchestration layer. Key components include:

    • SparkETLApp[C]: The fastest entry point for common ETL jobs.
    • SparkApp[C]: Used for fully custom runtime flows.
    • AppContext[C]: A context object carrying the SparkSession, typed configuration, CLI passthrough arguments, and a logger.
    • YAML Configuration: Support for decoding YAML files directly into Scala case classes.
    • Simplified IO: Easy data access via ctx.readInput(...) and ctx.writeOutput(...).
    • DataFrameUtils: Provides implicit DataFrame extensions for common operations.

  3. Implement data engineering patterns with SparkPlusPlus

    SparkPlusPlus jobs follow a consistent framework structure regardless of complexity. A standard application consists of:

    • YAML Configuration: Definition of inputs and outputs.
    • Entrypoint: Use of the SparkETLApp class.
    • Session Management: Spark session settings defined via sparkConfig.
    • Business Logic: Application logic centered on joins, transformations, and returning named outputs.

  4. Choose between SparkApp and SparkETLApp entry points

    SparkPlusPlus provides two application entry points depending on the level of control needed over the runtime:

    1. SparkApp[C]: Used for custom runtime control. You must implement appName, configClass, and the run(ctx) method.
    2. SparkETLApp[C]: Used for framework-managed ETL jobs. You typically implement appName, configClass, and the transform(ctx, inputs) method.

  5. Understand the Spark++ testing model

    The Spark++ repository employs a two-tier testing strategy to separate framework logic from Spark-dependent functionality:

    • Pure Unit Tests: Used to verify SparkApp argument parsing, configuration loading, and lifecycle behavior.
    • Spark Integration Tests: Used to cover DataFrameUtils and other Spark-backed functionality.

  6. Use SparkPlusPlus DataFrame utility helpers

    SparkPlusPlus provides a utility layer for common DataFrame tasks to simplify repetitive transformations. Available helpers include:

    • dedup: Remove duplicate records.
    • addRowNumber: Add a sequential row number.
    • countNulls: Count null values in columns.
    • getBasicStats: Retrieve basic statistics.
    • renameColumns: Rename DataFrame columns.
    • flattenFields: Flatten nested fields.
    • makeColumnNamesAvroCompliant: Ensure column names meet Avro compliance standards.

  7. Standardize Spark batch applications with SparkApp

    Use the SparkApp component to standardize the lifecycle of batch applications. It provides a consistent approach to configuration loading, SparkSession creation, logging, and application shutdown.

  8. Understand the SparkPlusPlus release and publishing flow

    SparkPlusPlus utilizes GitHub Actions to manage its release lifecycle:

    • CI/CD: Maven builds and tests are triggered automatically on pushes and pull requests.
    • Publishing: Artifacts are published to Sonatype when a tag is pushed to the repository.
    • Artifact ID Handling: The root pom.xml maintains a constant baseline artifact ID for Central publishing safety. For Scala 2.13, the artifact ID is swapped during the release workflow. Detailed publishing configurations are located in the dev/ directory of the repository.

  9. Implement a SparkETLApp for framework-managed ETL

    When using SparkETLApp, the framework orchestrates the data flow.

    Required Implementations:

    • appName: The name of the application.
    • configClass: The class of the configuration object.
    • transform(ctx, inputs): The core logic that takes input DataFrames and returns a map of output DataFrames.

    Optional Overrides:

    • validateConfig(config): Custom validation for the configuration.
    • configureSpark(builder, config): Customize the SparkSession.Builder before the session is created.
    • beforeSparkStart(config, args, logger): Logic to execute before the Spark session starts.
    • extract(ctx): Custom extraction logic.
    • load(ctx, outputs): Custom loading logic.
    object OrdersJob extends SparkETLApp[OrdersConfig] {
  override protected def appName: String = "orders-job"
  override protected def configClass: Class[OrdersConfig] = classOf[OrdersConfig]

  override protected def transform(
    ctx: AppContext[OrdersConfig],
    inputs: Map[String, DataFrame]
  ): Map[String, DataFrame] = {
    // Transformation logic here
    Map("customer_orders" -> resultDataFrame)
  }
}

  10. Set up the SparkPlusPlus documentation site locally

    The documentation site is managed via npm. Use the following commands to install dependencies, start the local development server, or validate a production build.
    npm install
npm run start

# To validate a production build
npm run build

  11. Choose between SparkPlusPlus customer order examples

    Depending on the project stage, choose between two implementation patterns for customer order processing:

    • End-to-End Customer Orders: The recommended starting point. It demonstrates the full data flow using the smallest possible functional application.
    • Advanced Customer Orders: Designed for production-style jobs. It extends the basic flow by adding schema-controlled inputs and additional output cleanup processes.

  12. Maintain testing and documentation standards

    Contributors must adhere to the following quality expectations:

    • Unit Tests: Add or update unit tests whenever framework behavior changes.
    • Integration Tests: Keep Spark-backed integration tests separate from pure framework tests (this prevents failures in restricted environments where Spark cannot bind ports).
    • Documentation: If a public API or configuration shape changes, update the main docs in /docs/, relevant use-case examples in /use-cases/, and any outdated README examples.