DOCSEARCHbyAlgolia
Submit Docs

Apache Hive Documentation

repository·master·Indexed 26 days ago

https://github.com/apache/hive

Data warehouse software built on top of Apache Hadoop for reading, writing, and managing large datasets using SQL. This documentation covers integration with Apache HBase and Kafka via specialized storage handlers, including SerDe configuration, SSL setup, and ETL pipeline patterns. It also provides guides for running Hive in Docker, configuring external Postgres metastores, and building Docker images from source.

Tokens
52.1K
Snippets
72
Records
236
Agent score
91%

What's inside Apache Hive

  1. Overview of Apache Hive

    master

    Apache Hive is a data warehouse software built on top of Apache Hadoop that facilitates reading, writing, and managing large datasets using SQL. It is designed for large-scale data warehousing tasks like ETL, reporting, and data analysis rather than online transaction processing (OLTP).

    Key capabilities include:

    • SQL Access: Enables data warehousing via SQL, including OLAP functions, subqueries, and common table expressions.
    • Data Structuring: Imposes structure on various data formats.
    • Storage Integration: Accesses files in Apache HDFS or other systems like Apache HBase.
    • Query Execution: Uses the Apache Tez framework for interactive query execution with reduced overhead compared to MapReduce.
    • Extensibility: Supports user-defined functions (UDFs), user-defined aggregates (UDAFs), and user-defined table functions (UDTFs).

  2. Overview of Apache HCatalog

    master

    Apache HCatalog is a table and storage management service designed for data created using Apache Hadoop. It provides a management layer that includes:

    • Shared Schema and Data Types: A common mechanism for describing data.
    • Table Abstraction: Allows users to interact with data in a tabular format without needing to manage the underlying storage location or format.
    • Interoperability: Enables different data processing tools—such as Pig, MapReduce, Streaming, and Hive—to share and access the same data seamlessly.

    HCatalog acts as a single input/output interface, allowing users to choose the best tool for their specific use case (e.g., Pig vs. Hive) while ensuring they can still access shared datasets on the Hadoop grid.

  3. Overview of Hive Kubernetes Operator

    master

    The Hive Kubernetes Operator is a Java-based operator that manages Apache Hive clusters declaratively using a single HiveCluster custom resource. It is built using the Java Operator SDK (JOSDK) and the fabric8 Kubernetes client.

    Key Features

    • Single CRD: Uses HiveCluster to manage all Hive components.
    • Managed Services: Manages Metastore, HiveServer2, LLAP, and Tez AM.
    • Automatic Dependency Ordering: Handles the sequence of schema initialization $\rightarrow$ Metastore $\rightarrow$ HiveServer2 $\rightarrow$ LLAP/TezAM.
    • Storage Agnostic: Supports S3A, ABFS, GCS, HDFS, and Ozone.
    • High Availability: The Helm chart provides a Full-HA cluster configuration by default.
    • Status Reporting: Tracks readiness for each component within the CRD status.

  4. Use the HBase Storage Handler for Hive integration

    master
    The HBase Storage Handler allows Apache Hive to interact with data stored in Apache HBase. You can use this handler to create Hive tables that map directly to HBase tables, enabling SQL-based querying of HBase data. For detailed configuration steps, table creation syntax, and integration best practices, refer to the official Apache Hive documentation.

  5. Understand Hive Kubernetes Operator Architecture

    master

    The Hive Kubernetes Operator manages resources via a HiveClusterReconciler.

    Core Components (JOSDK Workflow Dependents)

    • HadoopConfigMapDependent (core-site.xml)
    • MetastoreConfigMapDependent (metastore-site.xml)
    • HiveServer2ConfigMapDependent (hive-site.xml + tez-site.xml)
    • SchemaInitJobDependent (schematool -initOrUpgradeSchema)
    • MetastoreDeploymentDependent $\rightarrow$ MetastoreServiceDependent
    • HiveServer2DeploymentDependent $\rightarrow$ HiveServer2ServiceDependent
    • ScratchPvcDependent (shared scratch PVC for HS2 $\leftrightarrow$ TezAM)

    LLAP and TezAM Management

    LLAP clusters and their paired TezAM instances are managed imperatively. For each entry in llapClusters[] in the CR spec, the operator produces:

    • LLAP: StatefulSet ({cluster}-{name}), headless Service, ConfigMap (llap-daemon-site.xml), and PDB.
    • TezAM: Deployment ({cluster}-tezam-{name}), headless Service, and ConfigMap (tez-site.xml).

    These are applied via serverSideApply() and are automatically garbage-collected using label-based discovery when removed from the CR.

    Startup Order

    1. ConfigMaps (Hadoop, Metastore, HiveServer2)
    2. Schema Init Job (if Metastore is enabled)
    3. Metastore Deployment + Service (if enabled)
    4. HiveServer2 Deployment + Service + Shared Scratch PVC
    5. LLAP clusters + paired TezAM instances (if enabled)

  6. Understand the Apache Hive Security Model

    master

    Apache Hive is a SQL data-warehouse layer over Apache Hadoop. It is not a standalone secured appliance; it is a clustered service that relies on an operator-controlled perimeter and trusted dependencies like Hadoop (HDFS, YARN), a metastore RDBMS, an external authorization provider (e.g., Apache Ranger), and a KDC for Kerberos.

    Core Components & Surfaces:

    • HiveServer2 (HS2): The SQL entry point. Accepts queries via Thrift/binary or HTTP. It handles authentication, compilation, authorization, and execution. This is the primary untrusted network boundary.
    • Hive Metastore (HMS): A Thrift service managing table, partition, and schema metadata. It must enforce application-level authorization because external services (like Apache Spark) may connect to it directly.
    • Query Compiler & Execution: Parses and runs queries on engines like Tez, MapReduce, or Spark, interacting with HDFS, HBase, or object stores.
    • UDF / SerDe / File-format Layer: In-JVM code execution via user-supplied or built-in functions and (de)serializers.

  7. Quickstart: Run Apache Hive in Docker (Pseudo-distributed mode)

    master
    You can run Apache Hive inside a Docker container in pseudo-distributed mode. This is useful for quick-starts, debugging, or preparing a test environment for Hive. The setup involves pulling the image, identifying the Hive version, launching HiveServer2 with an embedded Metastore, and connecting via Beeline.

  8. Quickstart: Setup Hive, Gravitino, and Keycloak with Docker Compose

    master

    This setup integrates Apache Hive, Gravitino (Iceberg REST server), and Keycloak (OAuth2) using Docker Compose. It allows Hive to use an Iceberg REST catalog secured via Keycloak.

    Prerequisites

    • Docker & Docker Compose
    • Java (for local Hive beeline client)
    • $HIVE_HOME environment variable pointing to your Hive installation.

    Setup Steps

    1. Export the Hive version to ensure the correct image is used.
    2. Start the services in detached mode.
    3. Connect via Beeline to verify the connection.
    4. Stop services and clean up volumes when finished.
    # STEP 1: Export the Hive version
export HIVE_VERSION=4.2.0

# STEP 2: Start services
docker-compose up -d

# STEP 3: Connect to beeline
"${HIVE_HOME}/bin/beeline" -u "jdbc:hive2://localhost:10001/default" -n hive -p hive

# STEP 4: Stop services
docker-compose down -v

  9. Quickstart: Run Hive Standalone Metastore with Docker

    master

    To run a quick instance of the Apache Hive Metastore using Docker with a Derby backend (note: data is not persistent in this mode), follow these steps:

    1. Pull the specific versioned image from DockerHub.
    2. Export the HIVE_VERSION environment variable.
    3. Run the container mapping port 9083.

    This is suitable for testing but not for production as all data is lost when the service stops.

  10. Quickstart: Launch HiveServer2 with embedded Metastore

    master

    For a lightweight, quick setup, you can run HiveServer2 with an embedded Derby metastore using Docker. This is suitable for testing but data is not persistent across container restarts.

    1. Pull the image:

    docker pull apache/hive:4.0.0 2. Set the version environment variable: export HIVE_VERSION=4.0.0 3. Run the container: docker run -d -p 10000:10000 -p 10002:10002 --env SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION} 4. Connect via Beeline: docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'

    docker pull apache/hive:4.0.0
export HIVE_VERSION=4.0.0
docker run -d -p 10000:10000 -p 10002:10002 --env SERVICE_NAME=hiveserver2 --name hive4 apache/hive:${HIVE_VERSION}
docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'

  11. Configure SSL for Kafka Storage Handler

    master

    You can establish secure SSL connections to Kafka by using properties that reference a credential store. This avoids storing plaintext passwords in table properties.

    To use SSL:

    1. Provide the credential store location via hive.kafka.ssl.credential.keystore.
    2. Provide keys for passwords (e.g., hive.kafka.ssl.keystore.password) which point to entries in the credential store, rather than the passwords themselves.
    3. Set kafka.consumer.security.protocol to SSL.
    4. Specify the location of the truststore and keystore using hive.kafka.ssl.truststore.location and hive.kafka.ssl.keystore.location. These can be HDFS paths; the handler will pull them locally for jobs.

    Note: Both producer and consumer use the same property names for SSL locations (e.g., hive.kafka.ssl.truststore.location).

    CREATE EXTERNAL TABLE 
  kafka_ssl ( 
    `data` STRING 
  )
STORED BY 
  'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
TBLPROPERTIES ( 
  "kafka.topic" = "test-topic",
  "kafka.bootstrap.servers" = 'localhost:9093',
  'hive.kafka.ssl.credential.keystore'='jceks://hdfs/tmp/test.jceks',
  'hive.kafka.ssl.keystore.password'='keystore.password',
  'hive.kafka.ssl.truststore.password'='truststore.password',
  'kafka.consumer.security.protocol'='SSL',
  'hive.kafka.ssl.keystore.location'='hdfs://cluster/tmp/keystore.jks',
  'hive.kafka.ssl.truststore.location'='hdfs://cluster/tmp/keystore.jks'
);

  12. Run Hive Metastore with S3-backed warehouse storage

    master

    To use S3 for warehouse storage, you must use docker-compose and provide the AWS SDK bundle.

    Requirements:

    • Hadoop 3.4.1 requires AWS SDK v2.
    • Download the AWS SDK bundle (e.g., bundle-2.42.25.jar) and place it in a jars/ directory.

    Required Environment Variables:

    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • DEFAULT_FS (e.g., s3a://your-bucket)
    • HIVE_WAREHOUSE_PATH
    • S3_ENDPOINT_URL
    # 1. Download AWS SDK bundle
wget https://repo1.maven.org/maven2/software/amazon/awssdk/bundle/2.42.25/bundle-2.42.25.jar -P jars/

# 2. Launch via docker-compose
DEFAULT_FS="s3a://dw-team-bucket" \
HIVE_WAREHOUSE_PATH="/data/warehouse/tablespace/managed/hive" \
S3_ENDPOINT_URL="s3.us-west-2.amazonaws.com" \
docker-compose up