DOCSEARCHbyAlgolia
Submit Docs

Apache Hadoop

repository·trunk·Indexed 12 days ago

https://github.com/apache/hadoop

A framework for distributed storage and processing, featuring the YARN Capacity Scheduler UI, the YARN Application Catalog for Docker-enabled cloud applications, and a Resource Estimator Service for predicting job resource requirements. Includes developer tools for build environment management via Docker, Git/Jira fixVersion validation, and patch management using Apache Yetus.

Tokens
361.7K
Snippets
835
Records
1.4K
Agent score
90%

What's inside Hadoop

  1. Overview of HttpFS for HDFS access

    trunk

    HttpFS is a REST HTTP gateway for HDFS that provides full filesystem read and write capabilities. It serves as an interface for interacting with HDFS using standard HTTP protocols instead of the native HDFS RPC protocol.

    Key use cases include:

    • Cross-version data transfer: Facilitates moving data between Hadoop clusters running different versions (e.g., via Hadoop DistCP) by overcoming RPC versioning incompatibilities.
    • Firewall traversal: Acts as a secure gateway for clusters located behind a firewall; the HttpFS server is the only component permitted to cross the firewall into the HDFS cluster.
    • Language and tool interoperability: Enables HDFS access using standard HTTP utilities like curl and wget, or via HTTP libraries in languages other than Java (e.g., Perl).

  2. Overview of the YARN Service API

    trunk
    The YARN Service API is a simplified REST API designed to create and manage the lifecycle of containerized services (such as Docker containers) on YARN. It uses a JSON specification (often referred to as a YarnFile) that can be used via both the REST API and the CLI. This API abstracts the complexities of native YARN or programmatic APIs, allowing application owners to deploy services without significant changes to their applications.

  3. Overview of the Resource Estimator Service

    trunk

    The Resource Estimator Service is a tool designed to parse history logs from production jobs to extract resource consumption skylines from past runs. It uses these patterns to predict the resource requirements for new job runs.

    Currently, the service supports:

    • Hadoop YARN ResourceManager logs parsing.
    • In-memory storage for parsed history resource skylines and estimations.
    • Linear Programming-based estimation (utilizing the ojAlgo library).
    • REST interface for parsing logs, querying the history store, and retrieving estimations.

  4. Overview of Dynamometer

    trunk

    Dynamometer is a performance testing tool for the Hadoop HDFS NameNode. It simulates a real-world HDFS cluster (referred to as Dyno-HDFS, Dyno-NN, and Dyno-DN) by launching a YARN application that starts a single NameNode and a configurable number of DataNodes.

    It allows you to replay production workloads by using actual production file system images and audit logs. This enables testing configuration changes or new Hadoop versions at scale without needing a physical large-scale cluster.

  6. Overview of the hadoop-aliyun module

    trunk

    The hadoop-aliyun module enables Apache Hadoop to integrate with Aliyun Object Storage Service (Aliyun OSS). It implements the standard Hadoop FileSystem interface, allowing Aliyun OSS to be used as a source or sink for data in MapReduce jobs and providing a hierarchical file system view of object storage.

    To include this module in the default Hadoop classpath, add hadoop-aliyun to the HADOOP_OPTIONAL_TOOLS list in your hadoop-env.sh file.

  7. Overview of YARN Capacity Scheduler UI

    trunk

    The YARN Capacity Scheduler UI is a web interface designed for managing Apache Hadoop YARN Capacity Scheduler configurations. It allows administrators to visually manage queue hierarchies, define placement rules, plan capacities, and stage configuration changes for validation before they are applied to a live cluster.

    Key Capabilities

    • Queue Management: Interactive tree visualization for creating, editing, and deleting queues, including capacity (percentage, weight, or absolute), ACLs, and resource limits.
    • Placement Rules: A guided editor for authoring and prioritizing rules based on users, groups, or application names.
    • Staged Changes: A system to review, compare, and validate configuration deltas in a unified panel before committing them to the cluster.
    • Node Labels & Partitions: Tools for managing resource partitions and assigning nodes to labels.
    • Validation: Real-time property-level and cross-queue dependency validation to prevent invalid configurations from being applied.

  8. Overview of the Hadoop S3A Client

    trunk

    The hadoop-aws module provides the S3A client, a high-performance IO connector for Amazon S3 and compatible object stores.

    Key Features:

    • High Performance: Supports partitioned uploads for large objects and high-performance random IO for columnar formats like Apache ORC and Parquet.
    • Compatibility: Compatible with standard S3 clients and Amazon EMR's s3:// (EMRFS) files.
    • Authentication: Supports environment variables, Hadoop configuration properties, Hadoop key management store, and IAM roles.
    • Encryption: Supports S3 Server-Side Encryption (SSE-S3, SSE-KMS, SSE-C) and S3-CSE client-side encryption.
    • Observability: Instrumented with Hadoop metrics.

  9. Overview of YARN Scheduler Load Simulator (SLS)

    trunk

    The YARN Scheduler Load Simulator (SLS) is a tool designed to simulate large-scale YARN clusters and application loads on a single machine. It allows researchers and developers to prototype and evaluate YARN scheduler algorithms (like FIFO, Capacity, or Fair schedulers) without the cost and complexity of a real cluster.

    Key Features:

    • Real Scheduler Execution: It exercises the actual YARN ResourceManager by simulating NodeManagers (NM) and ApplicationMasters (AM) via heartbeat events within the same JVM.
    • Workload Simulation: Supports real job traces (via Apache Rumen) or synthetic load distributions.
    • Metrics Generation: Produces real-time and offline metrics including cluster/queue resource usage, application execution traces, and scheduler operation time costs (e.g., allocate, handle).

  10. Overview of Hadoop NFS library components

    trunk

    Hadoop NFS is a Java library designed for building NFS (Network File System) gateways. It provides implementations for several core network protocols required for NFS functionality:

    • ONCRPC: An implementation of ONCRPC (RFC-5531) and XDR (RFC-4506).
    • Mount: An interface implementation of the MOUNT protocol (RFC-1813).
    • Portmap: An implementation of the Binding protocol (RFC-1833).
    • NFSv3: An interface implementation of the NFSv3 protocol (RFC-1813).

  12. Overview of HDFS NFS Gateway

    trunk

    The HDFS NFS Gateway supports NFSv3, allowing HDFS to be mounted as part of a client's local file system. This enables users to:

    • Browse the HDFS file system via the local file system (on NFSv3 compatible OS).
    • Download files from HDFS to the local file system.
    • Upload files from the local file system to HDFS.
    • Stream data directly to HDFS through the mount point.

    Note: File append is supported, but random write is not. The NFS gateway machine requires the same environment as an HDFS client (e.g., Hadoop JAR files and HADOOP_CONF_DIR).