Apache HugeGraph Documentation

HugeGraph PD (Placement Driver) is a meta server that provides cluster management and coordination for HugeGraph distributed deployments. It acts as the central control plane for:

• Service Discovery: Automatic registration and discovery of Store and Server nodes.
• Partition Management: Dynamic allocation, balancing, and rebalancing of partitions across Store nodes.
• Metadata Storage: Centralized storage of cluster metadata, configuration, and state.
• Node Scheduling: Intelligent scheduling and load balancing of graph operations.
• Health Monitoring: Continuous health checks and failure detection via heartbeats.

PD uses SOFA-jraft for Raft consensus and RocksDB for persistent metadata storage to ensure high availability and consistency.

The HugeGraph Store project is organized into several key modules:

• hg-store-common: Shared utilities and query abstractions (e.g., Condition, Aggregate, QueryCondition).
• hg-store-grpc: gRPC protocol definitions (proto files).
• hg-store-core: The core storage engine logic, including HgStoreEngine (singleton managing partition engines), PartitionEngine (wraps Raft nodes), and HgStoreStateMachine (implements JRaft's StateMachine).
• hg-store-client: Java client library providing HgStoreClient, HgStoreSession, and HgStoreQuery.
• hg-store-node: The Store node server (Spring Boot application).
• hg-store-cli: Command line interface.
• hg-store-dist: Distribution module.
• hg-store-test: Test suite.

The hugegraph-commons module provides several utility components designed to simplify the development of HugeGraph-related systems:

• Lock: Provides atomic locks, key locks, lock groups, and a lock manager.
• Config: Handles registration and loading of configuration options with built-in security checks.
• Event: Facilitates asynchronous listening and notification mechanisms.
• Iterator: Provides enhanced iterators with additional functions like map, filter, and extend.
• Rest: A RESTful client implemented on top of OkHttp, supporting POST, PUT, GET, and DELETE methods.
• Util: Includes performance analyzers, version checkers, numeric and collection utilities, and log/exception utilities.
• Rpc: An RPC component for internal module communication, currently based on Sofa-RPC.

HugeGraph Store is a distributed storage backend for HugeGraph designed for high availability, horizontal scalability, and strong consistency. It is built on RocksDB and Apache JRaft, serving as the data plane for large-scale graph workloads.

Key Use Cases:

• Production deployments: High availability (99.9%+ uptime) and workloads > 100GB.
• High concurrency: Scenarios requiring 1000+ QPS.
• Scalability: Environments needing horizontal expansion and fault tolerance.

When to use RocksDB Backend instead:

• Development and testing.
• Single-node deployments with moderate data (<100GB).
• Embedded scenarios where simplicity is preferred over distribution.

HugeGraph Server is composed of two primary layers:

1. Graph Engine Layer

• REST Server: Provides a RESTful API for querying graph/schema information. It supports Gremlin and Cypher query languages and includes APIs for service monitoring and operations.
• Graph Engine: Supports both OLTP (implementing the Apache TinkerPop3 framework) and OLAP graph computation types.
• Backend Interface: Manages the storage of graph data to the backend.

2. Storage Layer

• Storage Backends:
  • RocksDB: The default, embedded backend.
  • HStore: A distributed backend.
  • HBase: Deprecated and planned for removal in version 2.0.
  • Memory: A test-only backend.

Users can extend the system with custom backends without modifying existing source code.

HugeGraph Store uses partitions as the fundamental unit of data distribution.

• Partition: A logical unit of data identified by a unique id and a specific key range (hash values).
• Shard: A single replica of a partition residing on a specific Store node.
• Shard Group: The complete set of all shards (replicas) for a single partition, which together form a Raft consensus group.

Partition metadata includes the id, version, start_key, end_key, a list of shards, and the state (e.g., Normal, Offline).

The HugeGraph ecosystem includes several specialized components for data management, computing, and AI:

1. hugegraph-toolchain: A suite of graph tools including:
   • Loader: Data import tool.
   • Dashboard: Web visualization platform (Hubble).
   • Tool: Command-line utilities.
   • Client: Java/Python client SDKs.
2. hugegraph-computer: An integrated graph computing system.
3. hugegraph-ai: Integration for Graph AI, LLM, and Knowledge Graphs.
4. hugegraph-website: Documentation and official website.

HugeGraph PD is organized into a layered architecture consisting of 8 Maven modules:

• Client Layer: hg-pd-client (Java library) and hg-pd-cli (CLI tools).
• Service Layer: hg-pd-service (gRPC and REST/Spring Boot implementations).
• Core Layer: hg-pd-core (Raft consensus via JRaft, RocksDB metadata stores, partition allocation, and task coordination).
• Foundation Layer: hg-pd-grpc (Protobuf definitions) and hg-pd-common (Shared utilities).
• Distribution Layer: hg-pd-dist (Assembly, startup/shutdown scripts, and configuration templates) and hg-pd-test (Tests).