DOCSEARCHbyAlgolia
Submit Docs

Shifter Documentation

repository·master·Indexed 18 days ago

https://github.com/nersc/shifter

A container runtime designed for High-Performance Computing (HPC) environments to execute Docker images safely and efficiently. Shifter consists of a runtime utility, an Image Gateway service for pulling and repacking images (typically into squashfs), and integration scripts for batch schedulers. The documentation covers the shifter CLI, Image Gateway RESTful API authentication, and system administration best practices for kernel module configuration, loop device management, and mount hooks.

Tokens
20.3K
Snippets
56
Records
87
Agent score
62%

What's inside Shifter

  1. What is Shifter and how does it work?

    master

    Shifter is a tool designed to enable container images for High-Performance Computing (HPC) environments. It allows HPC systems to run Docker images efficiently and safely.

    Shifter's architecture consists of three main components:

    1. Runtime Utility: Typically runs on the compute node to create the runtime environment for the application.
    2. Image Gateway Service: Pulls images from a registry and repacks them into a format suitable for the HPC system (most commonly squashfs).
    3. Integration Scripts/Plugins: Provided to integrate Shifter with various batch scheduler systems.

  2. How siteFs bind-mounting works

    master

    The siteFs option allows you to automatically bind-mount specific host paths into the Shifter container. This is commonly used to make network filesystems (like GPFS or Lustre) accessible inside the container.

    Constraints and Best Practices:

    • Do not attempt to bind-mount under critical system paths like /usr or other common container paths.
    • Safe paths: It is recommended to bind-mount under /var, /opt, or a custom site-maintained path (e.g., /global).
    • Format: Provide a space-separated list of paths.

  3. Use Shifter sshd for complex workflows

    master

    If enable_sshd=1 is set in plugstack.conf, Shifter starts a statically linked sshd on port 204 within the container.

    • Access: It only allows logins using an automatically constructed SSH key for Shifter.
    • Configuration: Shifter automatically injects the necessary SSH configuration into the container's /etc/ssh/ssh_config to handle the port redirection.
    • Node Discovery: Inside the container, you can discover other nodes in the allocation by reading /var/hostslist (in PBS_NODES format).
    • Requirement: The sshd capability is only available in node-exclusive allocations. Shared allocations do not run setupRoot and therefore cannot start the sshd.

  4. Emulate Cray CCM in native Slurm

    master

    Shifter can emulate Cray Cluster Compatibility Mode (CCM) to allow a user's session to be automatically placed into a Shifter environment. This is useful for targeting the existing external environment rather than a user-specified one.

    Requirements for CCM:

    1. In plugstack.conf, set both enable_ccm=1 and enable_sshd=1 on the shifter_slurm.so line.
    2. In udiRoot.conf, set allowLocalChroot=1.

    When enabled, the user can use the --ccm flag with sbatch, salloc, or srun to launch the job in a containerized revision of the DSL environment.

  5. Use module hooks for container customization

    master

    Hooks allow you to execute scripts at specific stages of the container lifecycle. A non-zero exit status from a hook will terminate the Shifter process.

    • module_<name>_roothook: Executes a script with /bin/sh as root in the external environment. This runs after most site customization (like siteFs and copyPath) but before user content is introduced. The script's CWD is the container environment. Use this for environment transformations.
    • module_<name>_userhook: Executes a script with /bin/sh as the user after container construction, but immediately before the user's process is launched. This is useful for lightweight compatibility checks.

    Important for userhook: The hook runs after the container has lost access to the external environment. If you need to access files, ensure they are located in a path provided by module_<name>_copyPath or a siteFs mount.

  6. Security requirements for loop device filesystems

    master

    Shifter uses loop devices to optimize I/O performance. Because these devices allow the kernel to manage filesystems with privileged access, the underlying filesystem files must never be writable by users directly.

    Filesystems should be produced by trusted toolsets. Avoid directly importing squashfs filesystems unless the source is fully trusted, as they are handled with root privileges by the kernel.

  7. Understand shifter environment and loop device requirements

    master

    Shifter attempts to reuse existing compatible environments on a system to launch processes. If no compatible environment exists, it generates a new mount namespace and sets up a new environment. This allows multiple shifter instances to run simultaneously on the same node.

    Important Resource Note: Each shifter instance consumes at least one loop device. It is recommended that compute nodes have at least two available loop devices per expected shifter instance. (For example, NERSC allows up to 128 loop devices per compute node).

  8. What are Shifter Modules and how do they work?

    master

    Shifter modules allow system administrators to customize container behavior for specific sites. They provide a way to inject content, manipulate environment variables, and execute hooks.

    Modules support two types of hooks:

    1. Root hooks: Executed as root during container instantiation (outside the container).
    2. User hooks: Executed as the user inside the container immediately before the process starts.

    This system allows sites to provide local support for system capabilities (like specific MPI implementations or file systems) while giving users the choice to enable or disable these modifications. Users can disable all modules by using the --module=none flag.

  9. Configure authorized locations in privileged requests

    master

    When performing privileged requests, you provide a JSON document containing an authorized_locations map. This map allows the image gateway to authenticate with remote registries.

    Key Rules:

    • Identifier Matching: The keys in the authorized_locations hash must match the location identifiers defined in the image_manager.json configuration on the server.
    • The "default" location: Because a client may not know which specific remote location a request will be routed to, you can use a special "default" key. If the image manager's default location matches a specific name provided in your JSON, it will use the specific one. If no specific name is provided, it will fall back to the "default" credential provided in your request.
    • Credential Selection: You can provide a single credential or multiple credentials. The image manager is responsible for parsing the structure and selecting the most appropriate credential for the specific request.
    {
    "authorized_locations": {
        "default": "username:password",
        "registry-1.docker.io": "user:pass"
    }
}

  10. Security implications of siteFs mount flags

    master

    Shifter typically enforces security by mounting filesystems with the nosuid flag (ignoring setuid bits) and restricting the set of capabilities available to processes.

    Warning: Using the following mount flags bypasses these protections:

    • :rec (recursive bind mount): This copies mount flags from the base system and will not follow Shifter's nosuid standards.
    • :shared (shared mount propagation): This removes the mounts from Shifter's strict control.

    While privilege capability restrictions should still prevent escalation, these flags remove the nosuid layer of defense.

  11. Understand the Image Gateway architecture

    master

    The Image Gateway is composed of three main layers:

    1. API Layer: A thin RESTful translation layer that receives client requests.
    2. Manager Layer: Manages and tracks images using a MongoDB database. It uses Python threads to asynchronously spawn tasks for pulling, packing, and transferring images. Configuration is handled via imagemanager.json.
    3. Worker Layer: Performs low-level image manipulation (fetching, packing, transferring). Workers are separated from the manager to allow for task queuing (to prevent overloading) and remote execution. Configuration is also handled via imagemanager.json.

  12. Configure image selection for shifter

    master

    Shifter identifies which container image to use by checking sources in a specific order of precedence. You can set defaults via environment variables (useful for batch systems like Slurm) or override them using the --image command-line flag.

    Precedence Order:

    1. SHIFTER environment variable (contains both type and specifier).
    2. SHIFTER_IMAGE and SHIFTER_IMAGETYPE environment variables.
    3. SLURM_SPANK_SHIFTER_IMAGE and SLURM_SPANK_SHIFTER_IMAGETYPE environment variables.
    4. --image command line option.

    Image Format: The format for the --image option and the SHIFTER environment variable is: imageType:imageSpecifier

    • imageType: Typically docker, but can be other site-defined types.
    • imageSpecifier: For docker, this is typically the sha256 hash of the image manifest.

    Example:

    # Using the command line option
shifter --image docker:sha256:abcdef123456... my_command

# Using environment variables
export SHIFTER_IMAGE=sha256:abcdef123456...
export SHIFTER_IMAGETYPE=docker
shifter my_command
    shifter --image docker:sha256:abcdef123456... my_command