DOCSEARCHbyAlgolia
Submit Docs

Apache Ozone Documentation

repository·master·Indexed 22 days ago

https://github.com/apache/ozone

A scalable, redundant, and distributed object store for Hadoop and cloud-native environments like Kubernetes and YARN. It provides strong consistency via RAFT and supports S3 and Hadoop File System APIs. The project includes the HDDS/Ozone server framework, libo3fs (a JNI-based C API), and Ozone Recon. Documentation covers building from source via Maven, deploying with Docker Compose or Kubernetes, and interacting with the store using the AWS S3 CLI.

Tokens
213.8K
Snippets
373
Records
993
Agent score
78%

What's inside Apache Ozone

  1. Overview of HttpFS (HDFS over HTTP)

    master

    HttpFS is a REST HTTP gateway for HDFS that provides full filesystem read and write capabilities. It is designed to facilitate several specific use cases:

    • Cross-version Data Transfer: Enables transferring data between Hadoop clusters running different versions (e.g., using Hadoop DistCP) by overcoming RPC versioning issues.
    • Firewall Traversal: Acts as a gateway to access HDFS data in clusters located behind a firewall; the HttpFS server is the only component permitted to cross the firewall into the cluster.
    • Non-Java Client Access: Allows accessing HDFS data using standard HTTP utilities like curl and wget, or via HTTP libraries in languages other than Java (e.g., Perl).

  3. Overview of libo3fs

    master

    libo3fs is a JNI-based C API for the Ozone File System. It provides read and write functionality for OzoneFileSystem. It leverages libhdfs (a JNI-based C API for Hadoop's Distributed File System) to provide a subset of HDFS APIs for manipulating HDFS files and the filesystem.

    Key components:

    • libo3fs: The core C API.
    • libhdfs: Used for HDFS integration, typically found in $HADOOP_HDFS_HOME/lib/native/libhdfs.so.

  4. Overview of Recon API v1 endpoints

    master

    The Recon API v1 provides HTTP endpoints for monitoring, managing, and troubleshooting an Ozone cluster. It allows administrators to inspect cluster metadata, container statuses, and key management.

    Available Container-related Endpoints:

    • GET /api/v1/containers: List containers (Admin only).
    • GET /api/v1/containers/:id/keys: Retrieve keys for a specific container ID.
    • GET /api/v1/containers/missing: Find missing containers.
    • GET /api/v1/containers/quasiClosed: Find quasi-closed containers.
    • GET /api/v1/containers/:id/replicaHistory: Retrieve replica history for a specific container ID.
    • GET /api/v1/containers/unhealthy: List unhealthy containers.
    • GET /api/v1/containers/unhealthy/:state: List unhealthy containers filtered by state.
    • GET /api/v1/containers/unhealthy/export: Initiate an export of unhealthy container data.
    • POST /api/v1/containers/unhealthy/export: Initiate an export of unhealthy container data.
    • GET /api/v1/containers/unhealthy/export/:jobId: Check the status of an export job.
    • GET /api/v1/containers/unhealthy/export/:jobId/download: Download the results of an export job.

    For an interactive version of the API with detailed descriptions and example requests, use the Swagger UI.

  5. What is Apache Ozone?

    master
    Apache Ozone is a scalable, distributed object store designed for lakehouse workloads, AI/ML, and cloud-native applications. It is built to handle both small and large files, supporting deployments with billions of objects and exabytes of capacity. It provides strong consistency guarantees, multiple protocol interfaces (including S3 compatibility), and configurable durability options.

  6. Overview of Ozone Admin commands

    master

    The ozone admin command suite provides a collection of tools specifically designed for administrators to manage and debug the Ozone cluster. Key functional areas include:

    • Safemode Management: Check, enter, or exit safemode using ozone admin safemode. Use the --verbose flag to see the validation status of all rules used to evaluate the safemode state.
    • Container Debugging: Inspect, list, get, or create containers (the units of replication) using ozone admin container.
    • Pipeline Inspection: Check available pipelines (sets of datanodes) using ozone admin pipeline.
    • Datanode Information: Retrieve details about datanodes using ozone admin datanode.
    • Topology Visualization: Display rack-awareness and topology information using ozone admin printTopology.
    • Replication Management: Monitor replication status or start/stop replication in emergencies using ozone admin replicationmanager.
    • Ozone Manager (OM) Management: Access tools related to Ozone Manager High Availability (HA) and cluster information using ozone admin om.

    For detailed usage instructions for any specific subcommand, use the --help flag.

  9. About the Fault Injection Service

    master

    The Fault Injection Service is a tool designed to validate Apache Ozone under heavy stress and during system component failures. It operates independently of Ozone and provides a client tool to inject errors into the IO path.

    Current Capabilities:

    • Injecting delays on various filesystem interfaces.
    • Injecting specific failures on specific paths for specific operations.
    • Simulating temporary or on-disk data corruption on the IO path.
    • Resetting specific or all injected failures.

    Note: Future updates aim to add error injection capabilities to the network path.

  11. Ozone Manager Network Services and Persistent State

    master

    The Ozone Manager provides several network services for clients and management commands, and maintains state using RocksDB.

    Network Services

    • Namespace Operations: CRUD operations for Keys, Buckets, and Volumes.
    • Multipart Uploads: Support for large file uploads via multi-step processes (e.g., Initiate, Complete).
    • File System Calls: Optimized calls for hierarchical queries (e.g., GetFileStatus, CreateDirectory, CreateFile, LookupFile).
    • Security: ACL management (using internal ACLs) and authorization token operations (Get, Refresh, Cancel).
    • Management APIs: S3 access key management, ServiceList for service discovery, and DBUpdates for downloading snapshots (used by Recon).

    Persistent State (Stored in RocksDB)

    • Volume/Bucket/Key Tables: Metadata for the namespace hierarchy and block IDs.
    • OpenKey Table: Tracks keys that are created but not yet committed.
    • Authorization Token Table: Stores security-related tokens.
    • PrefixInfo Table: Stores directory-level ACL information for optimized hierarchical queries.
    • S3 Key Table: Manages S3 access keys.
    • Multipart Upload Info Table: Tracks ongoing multipart uploads.
    • Delete Table: Tracks blocks that need to be deleted from data nodes.

  12. Integrate applications with Apache Ozone

    master

    Apache Ozone supports integration with various applications through two primary interfaces:

    1. Hadoop-compatible OFS (Ozone File System) interface: Allows applications designed for Hadoop to interact with Ozone as a filesystem.
    2. S3 interface: Provides compatibility with the Amazon S3 API, allowing applications written for S3 to use Ozone as their storage backend.