DOCSEARCHbyAlgolia
Submit Docs

NVIDIA cuDF plugin for Apache Spark

repository·main·Indexed 21 days ago

https://github.com/nvidia/cudf-spark

The NVIDIA cuDF plugin for Apache Spark accelerates Spark workloads by offloading data processing to GPUs using the cuDF library. It serves as a drop-in replacement for standard Spark SQL operators. The repository includes tools for API validation between Spark and GPU Execs, a deterministic data generation tool (datagen) for scale testing, Delta Lake support across various Spark versions, and a host memory leak detection utility for libcudf.

Tokens
143.2K
Snippets
233
Records
526
Agent score
76%

What's inside cudf-spark

  1. Overview of NVIDIA cuDF plugin for Apache Spark

    main
    The NVIDIA cuDF plugin for Apache Spark (formerly known as the RAPIDS Accelerator for Apache Spark) is a plugin library that leverages GPUs to accelerate Spark processing using the cuDF (CUDA DataFrame) libraries. It aims to produce results that are bit-for-bit identical with standard Apache Spark.

  2. Supported Features in Release 23.02

    main

    Release 23.02 introduced several new features for GPU acceleration in Spark, including:

    • Hive Support: Support for scanning and writing Hive text tables (HiveTableScanExec).
    • Delta Lake: Support for UpdateCommand, DeleteCommand, and MERGE INTO operations on Databricks.
    • SQL Functions: Support for from_json (returning MapType), json_tuple, reverse (for strings and arrays), and batched full join.
    • Databricks Compatibility: Support for Databricks 11.3 ML LTS.
    • Profiling: New profiling tool section focused on I/O metrics.

  3. Performance Improvements in Release 23.04

    main

    Key performance optimizations in the 23.04 release include:

    • Regex Optimization: Optimized regexp_replace in multi-replace scenarios and improved regexp_extract by transpiling expressions to use only the necessary single capture group.
    • Delta Lake: Added support for Delta Lake optimized writes.
    • UCX: Upgraded and documented UCX 1.14 support.
    • Spilling Logic: Improved SpillableColumnarBatch to better inform spill code of actual batch usage.

  4. New features in Release 24.02

    main

    Release 24.02 introduced several enhancements to the cuDF Spark plugin, including:

    • JSON Support: Support for single quotes when reading JSON, mixed types as strings in GpuJsonToStruct, and non-default dateFormat in from_json.
    • Timezone & Timestamp Operations: Expanded support for non-UTC timezones across various functions including unix_timestamp, to_utc_timestamp, to_date, from_unixtime, and ParseToTimestamp. Support for specific timezones like CST (China Standard Time) and non-DST timezones was added.
    • SQL Functions: Added support for ascii, format_number, and improved regexp_extract with lazy quantifiers and specified group indices.
    • Compatibility: Added support for Spark 3.3.4 and Databricks 13.3. Support for RAPIDS Spark plugin on ARM architectures was also introduced.
    • Performance: Optimized row-based window operations for BOUNDED ranges and improved hash join logic to use the smaller table for the build side when the build side can change.

  5. Performance improvements in Release 24.08

    main

    The 24.08 release included the following performance optimizations:

    • GpuSubstringIndex: Reworked to use cudf::slice_strings.
    • RLIKE Optimization: Rewrote pattern1|pattern2|pattern3 to use multiple contains operations in rlike to improve execution speed.
    • Hash Aggregation: Explored hash aggregation passthrough on partial aggregates.

  6. Qualification and Profiling tools in Release 22.12

    main

    The 22.12 release enhanced the diagnostic and validation tools:

    • Databricks Integration: Created demo notebooks on Databricks for both the qualification tool and the profiler tool usage.
    • Qualification Tool Improvements: Added support for parsing expressions (part 2) and the ability to print timestamp-related functions.

  7. Performance Improvements in Release 25.06

    main

    Release 25.06 included several performance-related optimizations:

    • Centralized Priority Assignment: Introduced 'Zero Config' centralized priority assignment.
    • Aggregation Optimization: By default, the plugin now enables skipping aggregations when the inputs of the aggregation cannot be reduced.
    • LocalAggregate: Added a specific physical rule for LocalAggregate to improve efficiency.

  8. Performance improvements in Release 22.12

    main

    Release 22.12 included several performance-focused enhancements:

    • Z-Ordering: Added support for z-ordering acceleration.
    • Tiered Projections: Identified additional opportunities for tiered projections and improved efficiency by eliminating eclipsed columns in each tier.
    • Optimized Operators: Investigated new CUDF-like operators and more efficient bound checks for GpuCast.
    • Delta Lake Fallback: Implemented fallback to CPU for Delta lake delta_log Parquet checkpoint files.

  9. Features in Release 23.04

    main

    The 23.04 release introduced several new capabilities, including:

    • Delta Lake Support: Added support for Delta Lake auto compaction and optimized writes.
    • OOM Resilience: Implemented an Out-of-Memory (OOM) retry framework to improve job stability.
    • Alluxio Integration: Added support to expose the Alluxio master URL to support Kubernetes (K8s) environments.
    • Metrics: Added support for task-level metrics in the retry framework and removed executor-level spill metrics.
    • ML Support: Added support in tools event parsing for ML functions, libraries, and expressions.
    • Compatibility Changes: Dropped support for Databricks-9.1 ML LTS.

  10. New features in Release 24.06

    main

    Release 24.06 introduced several new capabilities and improvements:

    • Function Support: Added support for parse_url and ArrayFilter.
    • Observability: The plugin now dumps the complete set of build-info properties to the Spark eventLog via SparkRapidsBuildInfoEvent.
    • Performance: Optimized parse_url and specific RLIKE patterns (e.g., pattern[A-B]{X,Y}) using custom kernels.

  11. New features in Release 22.04

    main

    The 22.04 release introduced several key features for GPU-accelerated Spark processing, including:

    • Expanded Data Format Support:
      • Support for reading Avro (primitive types).
      • Improved JSON and CSV readers with better support for timestamps, dates, decimals, and boolean types.
      • Support for ORC forced positional evolution.
      • Support for spark.sql.legacy.timeParserPolicy when parsing CSV files.
    • Enhanced SQL Functions and Expressions:
      • Support for approx_percentile in reduction contexts.
      • Support for percent_rank.
      • Support for hypot SQL function.
      • Support for str_to_map (including regular expression delimiters).
      • Support for ArrayExists and GetArrayStructFields expressions.
      • Support for element_at with non-literal indices.
      • Support for casting map to string.
    • Improved Data Type Handling:
      • Support for filtering and comparisons on ANSI day time interval types.
      • Support for partitioning on Decimal 128 in <WindowExec>.
      • Support for pmod, remainder, and divide with decimal support.
    • Compatibility and Infrastructure:
      • Support for Databricks 10.4 ML LTS.
      • Added Spark 3.1.4 shim.
      • Set RMM async allocator as the default.
      • Dropped support for Spark 3.0.1, 3.0.2, 3.0.3, and Databricks 7.3 ML LTS.