DOCSEARCHbyAlgolia
Submit Docs

Apache HBase Documentation

repository·master·Indexed 26 days ago

https://github.com/apache/hbase

A distributed, versioned, column-oriented database modeled after Google's Bigtable and designed to run on top of Apache Hadoop. This documentation covers developer support tools, including Java code coverage with JaCoCo, Git/JIRA release auditing, Docker-based standalone builds and test clusters, Yetus precommit checks, API/ABI compatibility verification, and Maven archetypes for project generation.

Tokens
229.6K
Snippets
483
Records
1.3K
Agent score
90%

What's inside Apache HBase

  1. Overview of Apache HBase Coprocessors

    master

    HBase Coprocessors allow you to run custom code directly on the RegionServers where your data resides. This moves computation to the data, reducing network bottlenecks that occur when moving large amounts of data to a client for processing.

    Implementation Workflow

    1. Implement an Interface: Your class must implement one of the following interfaces:
      • Coprocessor
      • RegionObserver
      • CoprocessorService
    2. Load the Coprocessor: Load it either statically via configuration or dynamically using the HBase Shell.
    3. Invoke the Coprocessor: Call the coprocessor from your client-side code. HBase handles the execution transparently for Observers, while Endpoints require explicit invocation.
    WARNING

    Coprocessors are advanced features. Because they run directly on the RegionServer with direct data access, they can cause data corruption or malicious access. There is currently no resource isolation, so a misbehaving coprocessor can degrade cluster performance and stability.

  2. Overview of Git / JIRA Release Audit

    master

    The Git / JIRA Release Audit application performs an audit between git branch histories and the fixVersion field in JIRA issues.

    It works by:

    1. Building a Sqlite database from commits found on each git branch.
    2. Identifying Jira IDs and release tags within those commits.
    3. Requesting issue information from Jira.
    4. Allowing users to query the database to identify discrepancies between the git history and JIRA data.

  4. Key features of HBase

    master

    HBase is a distributed NoSQL data store designed for linear and modular scaling. Key features include:

    • Strongly consistent reads/writes: Unlike many NoSQL stores, HBase is not eventually consistent, making it suitable for high-speed counter aggregation.
    • Automatic sharding: Tables are distributed via regions that automatically split and redistribute as data grows.
    • Automatic RegionServer failover: Ensures availability during server failures.
    • Hadoop/HDFS Integration: Uses HDFS as its underlying distributed file system.
    • MapReduce Support: Enables massively parallelized processing using HBase as both a source and a sink.
    • Multiple Client APIs: Provides a Java Client API for programmatic access, as well as Thrift and REST APIs for non-Java environments.
    • Query Optimization: Utilizes Block Cache and Bloom Filters for high-volume query performance.
    • Operational Management: Includes built-in web pages for operational insight and JMX metrics.

  5. Use built-in coprocessor endpoints for aggregation and export

    master

    HBase provides standalone RPC services called coprocessor endpoints that can be deployed on region servers or masters. This module includes two primary implementations:

    1. AggregateImplementation: Performs server-side aggregations such as sum, min, max, and avg.
    2. Export: Performs server-side table exports directly to HDFS.

    To invoke these endpoints from a client application, use the helper classes located in the org.apache.hadoop.hbase.client.coprocessor package.

  6. Understand replication consistency and ordering

    master

    HBase replication provides at-least-once delivery of client edits.

    Important Considerations:

    • Ordering: Standard asynchronous replication does not guarantee the order of delivery for client edits. If a RegionServer fails, the recovery of the replication queue happens independently of the individual regions, which can lead to out-of-order delivery.
    • Idempotency: Because of at-least-once delivery and potential lack of ordering, applications using non-idempotent operations (e.g., Increments) may see inconsistent states across clusters.
    • Solution: Use Serial Replication if your application requires guaranteed order of client requests to be maintained at the destination cluster.

  7. Use Stripe Compactions for Large Regions

    master

    Stripe compactions (experimental) improve performance for large regions or non-uniformly distributed row keys by maintaining StoreFiles separately for row-key sub-ranges called "stripes". This reduces the scope of compactions and can improve read/write performance variability.

    When to use:

    • Large regions: Provides the benefits of smaller regions without the overhead of managing more regions.
    • Non-uniform keys: Only stripes receiving new keys need to compact, leaving old data untouched.

    Implementation Details:

    • Stripe compaction is compatible with ExploringCompactionPolicy or RatioBasedCompactionPolicy.
    • It changes the HFile layout to create sub-regions within regions.
    • It can be enabled for existing tables and disabled later without issues.

  8. Use HBaseContext for Spark and HBase integration

    master

    The HBaseContext is the core component for integrating Spark and HBase. It accepts HBase configurations and broadcasts them to Spark executors, allowing each executor to maintain a static HBase Connection. This enables Spark tasks to access a shared Connection object without requiring executors to be co-located with HBase Region Servers.

    Supported interaction points include:

    • Basic Spark: Using HBaseContext within a Spark DAG.
    • Spark Streaming: Using HBaseContext within Spark Streaming applications.
    • Spark Bulk Load: Writing directly to HBase HFiles for bulk insertion.
    • SparkSQL/DataFrames: Writing SparkSQL that interacts with HBase tables.

  9. Understand HBase Backup Terminology

    master

    The following terms are used to manage the backup and restore lifecycle:

    • A backup: A logical unit of data and metadata used to restore a table to a specific point in time.
    • Full backup: A backup that wholly encapsulates the contents of a table at a specific point in time.
    • Incremental backup: A backup containing only the changes in a table since the last full backup.
    • Backup set: A user-defined name referencing one or more tables for backup execution.
    • Backup ID: A unique identifier for a specific backup (e.g., backupId_1467823988425).

  11. Understand HBase Coprocessor Design (Composition vs Inheritance)

    master

    HBase is transitioning its Coprocessor design from an inheritance-based model to a composition-based model.

    Old Design (Inheritance): Observers (like RegionObserver) and Services (like CoprocessorService) directly extend the Coprocessor interface. This often forces developers to create massive classes that implement many different interfaces (e.g., MasterObserver, RegionObserver, EndpointObserver) in a single file.

    New Design (Composition): Instead of being a Coprocessor, an Observer is something a Coprocessor has. A specialized Coprocessor interface (e.g., RegionCoprocessor) provides getter methods to retrieve specific observers or services (e.g., getRegionObserver(), getService()). This allows developers to break out different observer implementations into separate, smaller, and more manageable classes.