DataFlint OSS

The SqlFlow component uses a specific design system to communicate Spark execution details visually:

• Colors:
  • Primary color: #3f51b5 (Material Design Blue).
  • Performance indicators: Uses a Green → Yellow → Red spectrum to show relative duration.
  • Node types: Each type (input, output, transformation, etc.) is assigned a unique color.
• Node Structure: Nodes feature a clear hierarchy with distinct header, content, and footer sections.
• Visual Indicators:
  • Performance Bars: Color-coded bars showing relative duration.
  • Node Type Indicators: Visual dots indicating the specific node type.
  • Highlighting: An animated pulse effect is used for highlighted nodes.
• Layout: Uses a consistent 8px grid system for spacing and padding.

DataFlint OSS operates as a plugin installed on the Spark Driver and the Spark History Server.

It works by:

1. Exposing additional HTTP resources that provide metrics not natively available in the standard Spark UI.
2. Serving a modern Single Page Application (SPA) web-app that fetches data from Spark dynamically, allowing for real-time updates without manual page refreshes.

Before running the PySpark examples, you must build the appropriate plugin JAR using sbt. The scripts expect the JARs to be built in the following locations:

• Spark 3.x: pluginspark3/target/scala-2.12/spark_2.12-0.8.6.jar
• Spark 4.x: pluginspark4/target/scala-2.13/spark_2.13-0.8.6.jar

# For Spark 3.x
sbt pluginspark3/assembly

# For Spark 4.x
sbt pluginspark4/assembly

To use DataFlint OSS in a Scala project, add the appropriate dependency to your sbt configuration based on your Spark version, then configure the SparkSession to load the DataFlint plugin.

Dependency Selection

• Spark 3.X: Use io.dataflint %% spark
• Spark 4.X: Use io.dataflint %% dataflint-spark4

Plugin Configuration

You must set the spark.plugins configuration to io.dataflint.spark.SparkDataflintPlugin during SparkSession creation.

// For Spark 3.X
libraryDependencies += "io.dataflint" %% "spark" % "0.9.9"

// For Spark 4.X
libraryDependencies += "io.dataflint" %% "dataflint-spark4" % "0.9.9"

// Loading the plugin
val spark = SparkSession
    .builder()
    .config("spark.plugins", "io.dataflint.spark.SparkDataflintPlugin")
    .getOrCreate()

You can install DataFlint OSS without changing your application code by passing the package and plugin configuration directly to the spark-submit command.

For Spark 3.X

spark-submit \
--packages io.dataflint:spark_2.12:0.9.9 \
--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin

For Spark 4.X

spark-submit \
--packages io.dataflint:dataflint-spark4_2.13:0.9.9 \
--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin

You can quickly deploy a Spark History Server that includes the DataFlint plugin and UI by building and running a Docker image. This method downloads the necessary JARs from Maven Central automatically.

1. Build the image: docker build -t dataflint-history-server .

2. Run the container: Mount your Spark event logs directory to /spark-history inside the container and map port 18080.

Access the server at http://localhost:18080. Once running, click on any Spark application to see the DataFlint tab.

# Build the image (downloads JAR from Maven Central)
docker build -t dataflint-history-server .

# Run with your event logs directory
docker run -d \
  -p 18080:18080 \
  -v /path/to/spark-events:/spark-history:ro \
  --name dataflint-history-server \
  dataflint-history-server

To use Docker Compose, set the SPARK_HISTORY_DIR environment variable to point to your host's Spark event logs directory, then start the service.

# Set your event logs directory
export SPARK_HISTORY_DIR=/path/to/spark-events

# Start the service
docker-compose up -d

When upgrading to the improved SqlFlow component, note the following breaking changes and compatibility details:

Breaking Changes

• Node Dimensions: Node sizes have increased from 280x280 to 320x300.
• Styling: CSS class names have been updated to use the new design system.
• API: Component APIs have changed to provide better TypeScript type safety.

Backward Compatibility

• All existing functionality is preserved.
• Previous URL parameters remain supported.
• Existing data structures remain compatible.

To ensure DataFlint can process your application data, you must enable event logging in your Spark applications and point them to a specific directory.

# Via SparkSession
spark = SparkSession.builder \
    .config("spark.eventLog.enabled", "true") \
    .config("spark.eventLog.dir", "/path/to/spark-events") \
    .getOrCreate()

# Via spark-submit
spark-submit \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=/path/to/spark-events \
  your_app.py

Databricks Runtime 17.3+ uses javax.servlet instead of jakarta.servlet. To ensure compatibility, you must use a dedicated shaded artifact.

While the plugin class remains io.dataflint.spark.SparkDataflintPlugin, use the following coordinate:

Artifact: io.dataflint:dataflint-spark4-databricks_2.13

To use DataFlint OSS in a PySpark environment, add the required Maven packages and the plugin configuration to your SparkSession builder.

Configuration for Spark 3.X

Use the package io.dataflint:spark_2.12:0.9.9.

Configuration for Spark 4.X

Use the package io.dataflint:dataflint-spark4_2.13:0.9.9.

# For Spark 3.X
builder = pyspark.sql.SparkSession.builder \
    .config("spark.jars.packages", "io.dataflint:spark_2.12:0.9.9") \
    .config("spark.plugins", "io.dataflint.spark.SparkDataflintPlugin")

# For Spark 4.X
builder = pyspark.sql.SparkSession.builder \
    .config("spark.jars.packages", "io.dataflint:dataflint-spark4_2.13:0.9.9") \
    .config("spark.plugins", "io.dataflint.spark.SparkDataflintPlugin")