DOCSEARCHbyAlgolia
Submit Docs

Apache Spark

repository·master·Indexed 12 days ago

https://github.com/apache/spark

A unified analytics engine for large-scale data processing supporting SQL, streaming, machine learning, and graph processing. This documentation covers internal development guides including the proto-based configuration system, the Forward Secure Auth Protocol v2.0, standardized error reporting via SparkThrowable, and running Docker integration tests.

Tokens
400.1K
Snippets
996
Records
1.5K
Agent score
98%

What's inside Spark

  1. Overview of Evaluation Metrics in spark.mllib

    master
    The spark.mllib library provides a suite of metrics designed to evaluate the performance of machine learning models. These metrics are categorized based on the type of machine learning application being used, such as classification (binary, multiclass, or multilabel), regression, and clustering. Choosing the correct metric depends on your specific application requirements and the nature of your data.

  2. Overview of Spark SQL subprojects

    master

    Spark SQL provides support for executing relational queries using either SQL or the DataFrame/Dataset API. It is organized into five functional subprojects:

    • API (sql/api): Contains public types such as DataType and Row. This component is designed to be shared between the Catalyst engine and Spark Connect clients.
    • Catalyst (sql/catalyst): An implementation-agnostic framework used for manipulating trees of relational operators and expressions.
    • Execution (sql/core): The query planner and execution engine that translates Catalyst's logical query plans into Spark RDDs. It includes the SQLContext interface for executing SQL or LINQ statements against existing RDDs and Parquet files.
    • Hive Support (sql/hive): Provides extensions for HiveQL compatibility, access to the Hive Metastore via Hive SerDes, and wrappers for Hive UDFs, UDAFs, and UDTFs.
    • HiveServer and CLI support (sql/hive-thriftserver): Provides the backend for the SQL CLI (bin/spark-sql) and a HiveServer2-compatible server for JDBC/ODBC connectivity.

  3. Overview of Apache Spark capabilities

    master

    Apache Spark is a unified analytics engine designed for large-scale data processing. It utilizes an optimized engine that supports general computation graphs for data analysis. Spark provides high-level APIs in Scala, Java, Python, and R (Note: R is deprecated).

    Key high-level tools and libraries include:

    • Spark SQL: For SQL queries and DataFrame operations.
    • pandas API on Spark: For running pandas-compatible workloads on Spark.
    • MLlib: A library for machine learning.
    • GraphX: For graph processing.
    • Structured Streaming: For stream processing.

  4. Overview of SparkR

    master

    SparkR is an R package providing a lightweight frontend for Apache Spark. It allows you to use a distributed SparkDataFrame implementation for operations like selection, filtering, and aggregation on large datasets, similar to R data frames or dplyr. It also supports distributed machine learning via MLlib.

    Note: SparkR is deprecated from Apache Spark 4.0.0 and will be removed in a future version.

  5. Overview of GraphX

    master

    GraphX is a Spark component designed for graphs and graph-parallel computation. It extends the Spark RDD abstraction by introducing the Property Graph abstraction: a directed multigraph where each vertex and edge can have associated properties.

    GraphX provides several key capabilities for graph processing:

    • Fundamental Operators: Tools for structural manipulation (e.g., subgraph), joining data (e.g., joinVertices), and message aggregation (e.g., aggregateMessages).
    • Pregel API: An optimized variant of the Pregel message-passing model for iterative graph algorithms.
    • Algorithms and Builders: A collection of built-in graph algorithms and utilities to simplify common analytics tasks.

  6. Overview of Spark SQL

    master
    Spark SQL is the module within Apache Spark designed for working with structured data. It provides a programming abstraction that allows users to execute SQL queries and interact with structured data using a variety of methods, including standard SQL syntax, DataFrames, and Datasets. The Spark SQL engine supports ANSI SQL compliance and provides a rich set of data types, operators, and functions.

  7. Deploy Spark on a cluster

    master

    Spark can be deployed using several cluster managers:

    • Standalone Mode: The simplest way to deploy Spark on a private cluster without external dependencies.
    • Hadoop YARN: Deploying Spark on top of Hadoop's resource negotiator.
    • Kubernetes: Deploying Spark applications directly on Kubernetes (including support for the Spark Kubernetes Operator).

  8. PySpark Overview

    master

    PySpark is the Python API for Apache Spark, enabling real-time, large-scale distributed data processing. It allows Python users to leverage Spark's power for data analysis at any scale.

    PySpark supports several key modules:

    • Spark SQL and DataFrames: For working with structured data using SQL queries or Python DataFrames.
    • Pandas API on Spark: For scaling pandas workloads to distributed environments with minimal code changes.
    • Structured Streaming: A scalable, fault-tolerant engine for stream processing built on the Spark SQL engine.
    • Machine Learning (MLlib): A library of high-level APIs for building and tuning machine learning pipelines.
    • Spark Core and RDDs: The low-level execution engine providing Resilient Distributed Datasets (RDDs).
    • Declarative Pipelines (SDP): A framework for building reliable and maintainable ETL data pipelines.
    • Spark Connect: A client-server architecture for remote connectivity to Spark clusters.

  9. Decision Trees in spark.mllib (RDD-based API)

    master

    The spark.mllib library provides distributed decision tree implementations for both binary/multiclass classification and regression. It supports both continuous and categorical features and is designed to handle large-scale datasets by partitioning data by rows across a cluster.

    Key characteristics:

    • Interpretability: Easy to understand decision paths.
    • Feature Handling: Handles categorical features and does not require feature scaling.
    • Non-linearity: Capable of capturing non-linearities and feature interactions.
    • Scalability: Distributed training allows for millions of instances.

    For tree ensembles like Random Forests and Gradient-Boosted Trees, refer to the Ensembles guide.

  10. Logging in PySpark with PySparkLogger

    master

    The pyspark.logger module provides structured client-side logging for PySpark users. It uses the PySparkLogger class to emit logs in a structured JSON format, which includes timestamps, log levels, logger names, messages, and additional context. This structured approach is ideal for machine-readable logs and debugging complex PySpark applications.

    from pyspark.logger import PySparkLogger

logger = PySparkLogger.getLogger()
logger.info("Message content")

  11. What is Spark Connect?

    master

    Spark Connect is a client-server architecture (introduced in Spark 3.4) that decouples Spark client applications from the Spark cluster. This allows for remote connectivity, enabling Spark to be used from any application or environment.

    As of Spark 3.4, it provides:

    • PySpark: DataFrame API coverage.
    • Scala: DataFrame and Dataset API support.

  12. What is a Property Graph in GraphX

    master

    A property graph in GraphX is a directed multigraph where user-defined objects are attached to each vertex and edge.

    Key characteristics:

    • Directed Multigraph: Supports multiple parallel edges between the same source and destination vertices (e.g., representing different relationship types like 'friend' and 'colleague').
    • Vertex Identification: Each vertex is uniquely identified by a 64-bit long (VertexId).
    • Parameterization: The graph is parameterized over vertex type (VD) and edge type (ED).
    • Immutability: Like RDDs, property graphs are immutable and distributed. Operations produce new graphs, reusing unaffected parts of the original structure to optimize performance.
    • Optimization: GraphX optimizes memory usage by storing primitive data types (e.g., int, double) in specialized arrays.