HdfsCLI

HdfsCLI uses aliases to manage connections to different HDFS clusters. These are defined in a configuration file located at ~/.hdfscli.cfg by default, or at a custom path specified by the HDFSCLI_CONFIG environment variable.

Each alias is defined in a section named ALIAS.alias. A valid alias section must include a url option (the WebHDFS namenode URL, including protocol and port). You can also specify a client class to change the connection type and a root path for the cluster.

Supported client classes:

• hdfs.client.InsecureClient (default)
• hdfs.client.TokenClient (for Kerberos/Token-based auth)
• hdfs.ext.kerberos.KerberosClient (via Kerberos extension)

To support High Availability (HA) namenodes with WebHDFS, provide multiple URLs delimited by a semicolon (;) in the url option. Use the [global] section to set a default.alias.

[global]
default.alias = dev

[dev.alias]
url = http://dev.namenode:port
user = ann

[prod.alias]
url = http://prod.namenode:port
root = /jobs/

To use arbitrary client classes in the HdfsCLI, you must first make them discoverable by specifying their location in the [global] section of the HdfsCLI configuration file.

You can use two options to tell the CLI where to load clients from:

• autoload.modules: A comma-separated list of Python modules (must be on Python's path).
• autoload.paths: A comma-separated list of paths to Python files.

Example: making hdfs.ext.kerberos.KerberosClient available:

[global]
autoload.modules = hdfs.ext.kerberos

Example: loading a custom client from a specific file path and defining an alias:

[global]
autoload.paths = /etc/hdfscli.py

[prod.alias]
client = SecureClient
url = https://host:port
cert = /etc/server.crt, /etc/key

Note: Options passed to clients from the CLI (via hdfs.client.Client.from_options) are always passed as strings. You may need to implement parsing logic in your custom client's __init__ method.

The hdfscli entry point allows performing common HDFS operations. All commands accept an --alias argument to specify which cluster configuration to use.

Uploading and Downloading

• Upload: Use upload to send files or folders to HDFS. Use --threads to specify parallelism.
• Download: Use download to retrieve files or folders. Use --threads for parallelism.
• Overwrite: By default, HdfsCLI prevents overwriting existing paths. Use the --force option to overwrite.
• Streaming: Use - as a path argument to stream a single file's contents to/from standard out/in.

Interactive Shell

Running hdfscli --alias=YOUR_ALIAS (or just hdfscli if a default is set) opens an interactive Python shell (using IPython if available). The HDFS client is automatically available as the variable CLIENT.

# Write a single file to HDFS
hdfscli upload --alias=dev weights.json models/

# Read all files inside a folder from HDFS and store them locally
hdfscli download export/results/ "results-$(date +%F)"

# Read a file from HDFS and append its contents to a local log file using stdout
hdfscli download logs/1987-03-23.txt - >>logs

# Interactive shell usage
$ hdfscli --alias=dev
In [1]: CLIENT.list('data/')
In [2]: CLIENT.status('data/2.json')

The hdfs.client.Client.read, hdfs.client.Client.upload, and hdfs.client.Client.download methods accept a progress callback argument.

The callback function is invoked every chunk_size bytes and receives two arguments:

1. hdfs_path: The source path of the file currently being transferred.
2. nbytes: The number of bytes currently transferred for this file, or -1 to signal that the file transfer has finished.

Note: The write method does not expose a progress argument; instead, you can achieve similar functionality by passing a custom data generator.

Example implementation of a progress tracker:

from sys import stderr
from threading import Lock

class Progress(object):
    """Basic progress tracker callback."""
    def __init__(self):
        self._data = {}
        self._lock = Lock()

    def __call__(self, hdfs_path, nbytes):
        with self._lock:
            if nbytes >= 0:
                self._data[hdfs_path] = nbytes
            else:
                stderr.write('%s\n' % (sum(self._data.values()), ))