megfile

Aliases allow you to map a custom protocol name to a specific backend configuration. This is useful for multi-cloud setups (e.g., mapping tos:// to an S3-compatible backend).

Via CLI

# Configure a profile for Volcengine TOS
$ megfile config s3 accesskey secretkey \
    --addressing-style virtual \
    --endpoint-url https://tos-s3-cn-beijing.ivolces.com \
    --profile-name tos

# Create the alias
$ megfile config alias tos s3+tos

Via Environment Variables

You can define aliases using the MEGFILE_ALIASES__<NAME> environment variable. This is equivalent to an entry in the [alias] section of megfile.conf and takes precedence over the config file.

export MEGFILE_ALIASES__TOS=s3+tos
export MEGFILE_ALIASES__DATASET=s3+prod://bucket/prefix

Once defined, you can use the alias directly in paths: tos://bucket/key.

megfile uses a URI-style path format to identify the backend protocol. Bare POSIX paths are automatically treated as file://.

Supported protocols include:

• file:// or bare paths: Local File System
• s3:// (and aliases): S3/OSS-compatible object storage
• sftp://: SFTP
• http:// / https://: HTTP/HTTPS
• hdfs://: HDFS
• webdav://: WebDAV
• stdio://: Stdio

You can create custom aliases (e.g., tos://) using the CLI: megfile config alias <alias> <protocol>.

Megfile allows you to interact with different HDFS servers by defining profiles. Once a profile is configured (via CLI or environment variables), you can access that specific server using the URI scheme hdfs+profile_name://.

Example of syncing files between two different HDFS profiles:

from megfile import smart_sync

smart_sync('hdfs+profile1://path/to/file', 'hdfs+profile2://path/to/file')

The megfile glob functionality uses Unix shell-style patterns to find pathnames that match a specified pattern. This is useful for filtering files or directories in various filesystems (local, S3, etc.) supported by megfile.

|pattern|meaning|
|---|---|
|`*`|matches any characters but '/'|
|`**`|matches everything|
|`?`|matches any single character|
|`[seq]`|matches any character in seq|
|`[!seq]`|matches any char not in seq|
|`{seq1,seq2}`|matches seq1 or seq2|

You can configure HDFS settings using environment variables. Environment variables take precedence over configuration files.

For a default configuration, use the following variables:

• HDFS_USER: HDFS user
• HDFS_URL: The URL for the HDFS server. To support High Availability (HA) WebHDFS namenodes, delimit multiple URLs with a semicolon (;).
• HDFS_ROOT: The HDFS root directory used when providing relative paths.
• HDFS_TIMEOUT: Request timeout for the HDFS server.
• HDFS_TOKEN: HDFS token if required by the server.
• HDFS_CONFIG_PATH: Path to the HDFS config file (defaults to ~/.hdfscli.cfg).
• MEGFILE_HDFS_MAX_RETRY_TIMES: Maximum number of retries for HDFS requests when catching retryable errors (defaults to 10).

To configure multiple HDFS servers using profiles, prefix the variables with {PROFILE_NAME}__ (e.g., PROFILE1__HDFS_USER).

Install the base library using pip. To support specific backends like HDFS, WebDAV, or the CLI, you must install the corresponding extras.

• Base install: pip install megfile (includes Local FS, S3/OSS, HTTP, and Stdio)
• CLI support: pip install 'megfile[cli]'
• HDFS support: pip install 'megfile[hdfs]'
• WebDAV support: pip install 'megfile[webdav]'

megfile uses environment variables and configuration files to manage credentials and endpoints. Environment variables take precedence over configuration files.

For S3/OSS-compatible storage, use the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_ENDPOINT_URL (optional)
• OSS_ENDPOINT (optional)
• AWS_ENDPOINT_URL_S3 (optional)
• AWS_S3_ADDRESSING_STYLE (optional)

You can enable command-line auto-completion for megfile by generating the completion script for your specific shell and writing it to your shell's configuration file.

Use the megfile completion <your_shell> command. For example, to enable completion for zsh, run megfile completion zsh to write the necessary configuration to ~/.zshrc. After running the command, you must open a new shell session for the changes to take effect.

# Generate and write zsh completion to ~/.zshrc
megfile completion zsh

# After opening a new shell, you can use TAB for path completion
megfile ls [TAB]

To use HDFS protocols with megfile, you must install the HDFS extra dependencies using pip.

pip install 'megfile[hdfs]'

If you need to operate across multiple S3 servers with different credentials, you can use named profiles. You can access these via URI schemes like s3+profile_name://bucket/key.

Using the CLI to create a profile

megfile config s3 access_key secret_key --profile-name profile1

This creates a [profile1] section in your configuration file.

Using Environment Variables for profiles

Prefix the variable name with the profile name followed by double underscores (PROFILE_NAME__):

• PROFILE1__AWS_ACCESS_KEY_ID
• PROFILE1__AWS_SECRET_ACCESS_KEY
• PROFILE1__AWS_SESSION_TOKEN
• PROFILE1__OSS_ENDPOINT
• PROFILE1__AWS_S3_ADDRESSING_STYLE

Python Usage Example

from megfile import smart_sync

# Syncing between two different S3 profiles
smart_sync('s3+profile1://bucket/key', 's3+profile2://bucket/key')

Install the core megfile library via PyPI. Depending on your needs, you can install additional support for HDFS or WebDAV using extra dependencies.

# Core installation
pip3 install megfile

# With HDFS support
pip3 install 'megfile[hdfs]'

# With WebDAV support
pip3 install 'megfile[webdav]'