sparklearning Documentation

Apache Spark is a fast, general-purpose cluster computing platform that acts as a computational engine for scheduling, distributing, and monitoring applications across a cluster.

Core Components

• Spark Core: The foundation containing task scheduling, memory management, fault recovery, and the RDD API.
• Spark SQL: Enables executing SQL-like queries on Spark data using standard BI or visualization tools.
• Spark MLlib: A machine learning library for algorithms like clustering, regression, and classification.
• Spark Streaming: Used for processing real-time streaming data.
• Spark GraphX: An API for graph parallel computations (e.g., joinVertices, subgraph, aggregateMessages).

Shuffling is the process of rearranging data within a cluster between different stages. It is triggered by wide transformations, which require data to be redistributed across the cluster.

Common triggers for shuffling include:

• repartition
• Bykey operations (except counting)
• Joins (especially cross joins)
• Sorting
• Distinct
• GroupBy

The Driver is the central process that runs your main() function. It sits on a node in the cluster and is responsible for:

1. Maintaining information about the Spark Application.
2. Responding to user input/programs.
3. Analyzing, distributing, and scheduling work across Executors.

Driver Responsibilities:

• Prepares the SparkContext.
• Declares RDD operations using Transformations and Actions.
• Submits the serialized RDD graph to the master.

A Shuffle Sort Merge Join is the standard strategy for joining two large tables (Big table-to-big table). It consists of a sort phase (sorting each dataset by the join key) followed by a merge phase (iterating through keys to find matches).

Use this join type when:

• Each key within the two large datasets can be sorted and hashed to the same partition by Spark.
• You are performing equi-joins (joining based on matching keys).
• You want to prevent unnecessary Exchange and Sort operations by pre-optimizing data (e.g., via bucketing).

A DataFrame is a distributed collection of data organized into named columns.

Why use DataFrames over RDDs? When using RDDs, Spark only sees operations as opaque lambda expressions (e.g., Iterator[T]). Because the internal structure of the data is not visible to the engine, Spark cannot perform automatic optimizations. DataFrames provide a structured schema that allows the Spark optimizer to improve query execution.

Stream processing involves continuously incorporating unbounded data (data with no predetermined beginning or end) to calculate results. Common use cases include processing credit card transactions, IoT device data, and click streams.

Key Principle: When designing your application, ensure that your business logic produces identical results whether it is applied to a streaming data source or a batch data source.

Parquet is a columnar storage format used in Spark that offers several advantages over row-based storage:

• Columnar Storage: Loads only the required columns for a query, reducing I/O.
• Schema Storage: The schema is stored in the file footer.
• Predicate Pushdown: Allows for efficient data filtering at the storage level.
• Data Skipping: Enables skipping irrelevant data blocks.
• Space Efficiency: Does not waste space storing missing values.