DOCSEARCHbyAlgolia
Submit Docs

Apache Kyuubi Documentation

repository·master·Indexed 25 days ago

https://github.com/apache/kyuubi

Documentation for Apache Kyuubi, including guides for Kubernetes deployment via Helm, Docker Compose playground setup, and TPC-DS data generation and benchmarking. It covers Spark extensions such as Kyuubi Spark AuthZ for fine-grained security (column/row-level authorization and data masking) and Kyuubi Spark Lineage for SQL execution tracking. Additionally, it provides instructions for monitoring Kyuubi using Prometheus and Grafana.

Tokens
144.4K
Snippets
247
Records
651
Agent score
81%

What's inside Apache Kyuubi

  2. Overview of Kyuubi Server Side Extensions

    master

    Kyuubi supports server-side extensions that allow administrators to inject custom functionality into specific Kyuubi server modules. These extensions are used to customize or enhance the core capabilities of the server.

    Available extension types include:

    • Authentication: Customizing how users are authenticated.
    • Configuration: Modifying how configurations are handled.
    • Events: Intercepting or reacting to server events.
    • Applications: Extending application-specific logic.

  3. What is Kyuubi Beeline

    master

    Kyuubi Beeline is a Command Line Shell that uses a JDBC driver to connect to a Kyuubi server to execute queries. It is derived from Hive Beeline and supports most of its functionalities.

    Note: Kyuubi Beeline only supports "remote mode". It has removed support for "embedded mode" because that mode is coupled with Apache Hive implementation details.

  4. Overview of Kyuubi Clients and APIs

    master

    Kyuubi provides several ways for end-users to interact with the server without needing to manage server-side deployment. You can connect using standard database drivers or direct programmatic APIs:

    • Standard Database Drivers: Use JDBC or ODBC drivers to build database applications in your preferred programming language.
    • Direct APIs: Use REST or Thrift APIs for flexible, direct access to Kyuubi services.
    • Other Interfaces: Kyuubi also supports a CLI, Python clients, BI Tools, and a Web UI.

  5. What is Apache Kyuubi?

    master

    Apache Kyuubi is a distributed, multi-tenant gateway designed to provide serverless SQL capabilities on data warehouses and lakehouses. It acts as a pure SQL gateway using the Thrift JDBC/ODBC interface, allowing users to manipulate large-scale data using extensible Spark SQL engines.

    Key features include:

    • HiveServer2-compatible API: Provides a familiar interface for existing workloads.
    • Multi-tenant Spark Support: Enables resource isolation, data security, and high availability by decoupling the Kyuubi server from the Spark engines.
    • Serverless Spark Execution: Allows running Spark engines in a serverless manner via cluster managers like Hadoop YARN or Kubernetes.

  6. Use the Kyuubi Hive JDBC Module

    master
    The Kyuubi Hive JDBC Module provides a client for Kyuubi and Spark that implements the Hive JDBC interface. It is designed to be a well-supported client for interacting with Kyuubi engines. The module is available on Maven Central and can be used as a shaded JAR to avoid dependency conflicts.

  7. Explore available Kyuubi engine connectors

    master

    Kyuubi provides connectors for different engines to access data from various data sources. Depending on the engine you are using, you can access specific documentation for:

    • Spark: Connectors for the Spark engine.
    • Flink: Connectors for the Flink engine.
    • Hive: Connectors for the Hive engine.
    • Trino: Connectors for the Trino engine.

    If the connector you require is not listed, you can report a feature request via the Kyuubi issue tracking system.

  8. Use the Kyuubi Hive BeeLine Module

    master
    The Kyuubi Hive BeeLine module provides an enhanced Beeline client specifically optimized for Kyuubi. It improves the debugging experience by supporting the display of launch engine logs when establishing a KyuubiConnection (available since version v1.4.0-incubating).

  9. Quick Start Requirements for Kyuubi

    master

    To run a basic Kyuubi deployment, you need three primary components, all of which are JVM-based and require JAVA_HOME to be set:

    1. A Client: Used to connect and submit queries (e.g., kyuubi-beeline).
    2. A Server: Serves clients and manages engines (Kyuubi Server).
    3. An Engine: Instantiates query execution environments (e.g., Spark, Flink, Trino, Doris, or Hive).

    Supported Versions

    ComponentRoleVersion
    JavaJRE8, 11, 17
    KyuubiGateway/Beeline
    SparkEngine3.3 to 3.5, 4.0 to 4.2
    FlinkEngine1.17 to 1.20
    TrinoEngineN/A (use trino-client v411)
    DorisEngineN/A
    HiveEngine2.1-cdh6, 2.3, 3.1
    ZookeeperHA>=3.4.x

  10. Kyuubi Security Overview

    master

    Securing an Apache Kyuubi deployment requires implementing a multi-layered security approach. The core components of Kyuubi security include:

    • Authentication (authn): Verifying the identity of users and clients connecting to Kyuubi.
    • Authorization (authz): Controlling access to resources (such as engines, databases, or tables) based on the authenticated identity.
    • Encryption: Protecting data in transit and at rest to prevent unauthorized interception or access.

  11. Connect to data sources using the Kyuubi Spark SQL Query Engine

    master

    The Kyuubi Spark SQL Query Engine leverages Spark DataSource APIs (both V1 and V2) to access data from various sources.

    By default, Kyuubi provides access to Hive warehouses and supports common file formats including:

    • Parquet
    • ORC
    • JSON

    Kyuubi can also be integrated with third-party storage and table formats such as:

    • Hudi
    • Iceberg
    • Delta Lake
    • Kudu
    • Apache Paimon
    • HBase
    • Cassandra
    • TiDB

    For testing and benchmarking purposes, Kyuubi also provides sample data sources like TDC-DS and TPC-H.