DOCSEARCHbyAlgolia
Submit Docs

Sail Documentation

repository·main·Indexed 25 days ago

https://github.com/lakehq/sail

Sail is a high-performance, Rust-native replacement for Apache Spark that is compatible with the Spark Connect protocol. It allows users to run PySpark SQL and DataFrame code on a distributed engine. Sail supports table formats like Delta Lake and Apache Iceberg, and integrates with catalog providers including AWS Glue, Unity Catalog, Hive Metastore, and Microsoft OneLake. It provides native support for various storage backends such as AWS S3, Azure, Google Cloud Storage, and HDFS.

Tokens
57.5K
Snippets
144
Records
377
Agent score
86%

What's inside Sail

  1. Overview of Sail multimodal compute engine

    main

    Sail is an open-source multimodal compute engine designed for both single-host and distributed settings. It is built to unify three primary workload types:

    • Batch processing
    • Stream processing
    • Compute-intensive AI workloads

  3. Overview of Hive Metastore Thrift IDL in Sail

    main
    Sail uses a modified version of the Apache Hive Metastore Thrift IDL to facilitate communication with Hive Metastore services. The IDL is sourced from Apache Hive (branch 2.3) with minor edits to remove the fb303 dependency, making it more portable for the Sail ecosystem. The code generation logic for this IDL is based on hive_metastore_rs.

  6. Understand the Sail Query Planning process

    main

    Sail processes queries through multiple stages of syntactic and semantic analysis. It supports both SQL strings and Spark relations (via the Spark Connect protocol) by converting them into an intermediate representation called the Sail spec.

    Query Lifecycle Stages:

    1. Parsing & Conversion:
      • SQL strings are parsed via an in-house SQL parser (supporting Spark SQL features) into an AST, then converted to the Sail spec.
      • Spark relations are converted directly into the Sail spec.
    2. Logical Planning: The Sail spec is resolved into an Apache DataFusion logical plan using metadata from the catalog and function registry.
    3. Physical Planning: The logical plan is optimized and converted into a DataFusion physical plan, which undergoes further optimization. Sail uses custom extension nodes to support Spark-specific features like PySpark UDFs.
    4. Execution:
      • Local Mode: The optimized physical plan is executed directly.
      • Cluster Mode: The physical plan is split into stages at data shuffle boundaries. Tasks (representing partitions within a stage) are distributed to workers for execution.

  7. Supported Spark DataFrame API features in Sail

    main

    Sail supports a wide range of the Spark DataFrame API, covering most common data processing use cases.

    Supported Features

    • I/O: Reading (SparkSession.read) and Writing (DataFrame.write, DataFrame.writeTo()).
    • Result Collection: DataFrame.show(), DataFrame.collect(), and DataFrame.count().
    • Schema: DataFrame.printSchema().
    • Query Operations:
      • Projection: DataFrame.select(), DataFrame.selectExpr().
      • Column Operations: DataFrame.withColumn(), DataFrame.replace(), DataFrame.drop().
      • Filtering: DataFrame.filter().
      • Aggregation: DataFrame.agg(), DataFrame.groupBy().
      • Joins: DataFrame.join().
      • Set Operations: DataFrame.union(), DataFrame.intersect(), DataFrame.exceptAll().
      • Limit/Offset: DataFrame.offset(), DataFrame.limit().
      • Sorting: DataFrame.sort(), DataFrame.orderBy().
    • NA & Views: DataFrame.na and View management (e.g., DataFrame.createOrReplaceTempView()).
    • Python Integration: PySpark UDFs and PySpark UDTFs.

    Partially Supported or Unsupported Features

    • Structured Streaming (SparkSession.readStream): Under construction.
    • Statistics functions (DataFrame.stat): Under construction.
    • RDD Access (DataFrame.rdd): Not supported.

  8. Understand Sail execution modes: Local vs Cluster

    main

    Sail supports two primary execution modes that allow you to scale workloads from development to production with minimal friction:

    • Local Mode: Designed for ad-hoc data analysis and development. Sail runs as a single process where a local job runner executes the optimized physical plan using multiple threads to process data partitions in parallel across available CPU cores.
    • Cluster Mode: Designed for distributed data processing at scale. Sail forms a distributed system consisting of a Sail server and multiple Sail workers. A cluster job runner manages a driver that schedules a distributed physical plan composed of multiple stages, which are further broken down into tasks executed by workers.

  9. Supported Spark SQL Data Types, Literals, and Expressions in Sail

    main

    Sail provides high compatibility with Spark SQL syntax:

    • Literals: Supports all Spark SQL literal syntax.
    • Data Types: Supports all Spark SQL data types.
    • Expressions: Supports most Spark SQL expression syntax, including unary and binary operators, predicates, and CASE clauses. Most Spark SQL functions are also supported.

  10. Review Sail vs. Apache Spark Benchmark Results

    main

    Sail was benchmarked against Apache Spark using a derived TPC-H workload (22 queries including filters, joins, aggregations, and subqueries) on a 100 GB Parquet dataset.

    Key Performance Advantages:

    • Speed: Sail is significantly faster, showing a query speed-up of 43% to 727% compared to Spark.
    • Memory Efficiency: Sail uses substantially less peak memory (approx. 22 GB vs Spark's 54 GB) and releases it after query execution.
    • Disk I/O: Sail avoids shuffle spills to disk (0 GB disk write), whereas Spark spilled over 110 GB of temporary data.
    • Cost: Due to lower resource requirements, Sail can achieve up to 94% cost reduction by running on smaller, cheaper infrastructure.

  11. Understand Sail's core value propositions

    main

    Sail is a high-performance compute engine designed for modern cloud environments. It is built in Rust and uses Apache Arrow for columnar memory layout, providing several advantages over JVM-based engines like Apache Spark or Apache Flink:

    • Performance: Uses Apache Arrow for SIMD-supported vectorized operations and avoids JVM Garbage Collection (GC) overhead.
    • Memory Efficiency: Low memory footprint (starts in seconds, uses minimal memory when idle) and eliminates the need for JVM tuning.
    • Robustness: Leverages Rust's memory safety and ownership model to ensure safe concurrency and eliminate memory bugs.
    • Compatibility: Acts as a drop-in replacement for Spark SQL and the Spark DataFrame API by communicating via the Spark Connect protocol over gRPC.
    • Simplicity: Provides a unified environment for ad-hoc SQL, distributed batch jobs, and AI data preprocessing.

  12. Understand Sail's communication protocols

    main

    Sail utilizes a separation of concerns between the control plane and the data plane using different gRPC protocols:

    • Control Plane: Uses the internal Sail gRPC protocol for communication between the driver and workers. This is based on the actor model to manage state safely without locks. The driver and workers act as both gRPC servers and clients, but workers do not communicate with each other directly in the control plane.
    • Data Plane: Uses the Arrow Flight gRPC protocol for exchanging shuffle data among workers and returning results from workers to the driver.