SingularityCE Documentation

repository·main·Indexed 21 days ago

https://github.com/sylabs/singularity

A container platform optimized for High Performance Computing (HPC) and shared environments, focusing on security, mobility, and integration with host hardware like GPUs and parallel filesystems. This documentation provides detailed guides on developing and running end-to-end (e2e) tests, including the use of e2e.TestEnv, execution profiles (such as UserProfile and RootProfile), and the RunSingularity functional options for CLI testing.

Tokens
118.6K
Snippets
362
Records
487
Agent score
76%

What's inside SingularityCE

  1. Overview of SingularityCE

    main

    SingularityCE is an open-source container platform designed for ease-of-use on shared systems and High Performance Computing (HPC) environments. Unlike many other container platforms, it prioritizes integration over isolation, allowing seamless access to GPUs, high-speed networks, and parallel filesystems.

    Key features include:

    • SIF (Singularity Image Format): An immutable single-file container image format that supports cryptographic signatures and encryption.
    • Mobility: The SIF format makes containers easy to transport and share.
    • Security Model: Users remain the same user inside the container as they are outside, preventing unauthorized privilege escalation on the host system by default.
  2. Overview of the MAKEIT build system

    main
    MAKEIT is a lightweight, platform-independent build system designed to generate native, non-recursive Makefiles. It transforms Makefile fragments and module configuration files into Makefiles that are compatible with various versions of Make, including GNU, BSD, and SVR4. It is designed to be small enough to be included directly within individual projects.
  3. Configure requirements for OCI-mode execution

    main

    To use the --oci execution mode in SingularityCE, your system must provide one of the following:

    1. squashfs-tools / squashfs version 4.5 or higher (provides the sqfstar utility).
    2. squashfs-tools-ng (provides the tar2sqfs utility).

    Distribution-specific setup:

    • Debian/Ubuntu: squashfs-tools-ng is included in the standard dependency list provided in the system dependencies guide.
    • Fedora: squashfs-tools includes sqfstar by default.
    • RHEL / Alma Linux / Rocky Linux: Install squashfs-tools-ng via the EPEL repository. If you previously used the dctrud/squashfs-tools-ng COPR, remove it first:
    sudo dnf copr remove dctrud/squashfs-tools-ng
    # After enabling EPEL:
    sudo dnf install squashfs-tools-ng
    sudo dnf install squashfs-tools-ng
  4. Determine the license for files in the COPYING project

    main

    The COPYING project uses a dual-license model (BSD-3-Clause AND MPL-2.0). To determine which license applies to your usage, you must check the comment at the start of each individual file.

    In the event of a conflict between this documentation and the file-level comments, the comment at the start of the file takes precedence.

  5. Understand the e2e.TestEnv struct and initialization

    main

    The e2e.TestEnv struct is the central environment controller for end-to-end tests, initialized by e2e.Run() in e2e/suite.go.

    Key Initialization Features:

    • Isolated Filesystem: Creates a temporary TestDir. It sets up "fake" home directories for the current user and root by bind-mounting them in a dedicated mount namespace, ensuring tests don't affect real user files.
    • Default Configs: Populates TestDir with default versions of singularity.conf, remote.yaml, plugin directories, ECL configuration, and a global keyring.
    • Docker Authentication: If E2E_DOCKER_USERNAME and E2E_DOCKER_PASSWORD are set, it generates docker-config.json files in the fake home directories to avoid DockerHub rate limits.
    • Local Registry: Spins up a local Docker/OCI registry. The address (without the docker:// prefix) is available via testenv.TestRegistry.

    Pre-loaded Registry Images:

    • testenv.TestRegistryImage: A copy of docker://alpine:latest.
    • testenv.TestRegistryLayeredImage: A copy of docker://sylabsio/aufs-sanity:latest (for testing overlay/layers).
    • testenv.TestRegistryPrivURI: The base URI for the private testing repository.
    • testenv.TestRegistryPrivImage: A template for private images: testenv.TestRegistryPrivPath + "docker://%s/my-alpine:latest".
  6. Handle parallel vs sequential tests

    main

    The Singularity e2e suite runs tests in parallel by default. Most tests should be written to be thread-safe by using dedicated temporary directories under testenv.TestDir for all file operations.

    When to use Sequential (SEQ) tests: Some tests cannot run in parallel because they modify global state, such as:

    • Changing the current working directory.
    • Changing the OS umask.
    • Modifying files in the user's or root's home directory (e.g., $HOME/.singularity/remote.yaml).

    How to mark a test as sequential: Use testhelper.NoParallel(func(*testing.T)) to wrap a test function. This marks it to be run in the SEQ group rather than the PAR group.

    // Marking a test to run sequentially
    np := testhelper.NoParallel
    
    return testhelper.Tests{
        "use exclusive": np(c.remoteUseExclusive),
    }
  7. Understand Singularity e2e Profiles

    main

    The e2e suite uses profiles to represent different execution environments for Singularity. Profiles allow you to simulate running Singularity as root, as a regular user, with --fakeroot, or in OCI mode without manually configuring every flag.

    Profiles control:

    • Execution identity (root vs. regular user).
    • Default Current Working Directory (CWD).
    • CLI options (e.g., --fakeroot, --oci) and which commands they apply to.
    • OCI mode status.
    • Gating functions (to skip tests if host conditions aren't met).
    • Host and in-container UIDs.

    Commonly used profiles include:

    • e2e.UserProfile: Regular user, native runtime.
    • e2e.RootProfile: Root, native runtime.
    • e2e.FakerootProfile: Fakeroot, native runtime.
    • e2e.OCIUserProfile: Regular user, OCI mode.
    • e2e.OCIRootProfile: Root, OCI mode.

    Profiles are accessed via the e2e package (e.g., e2e.RootProfile). You can also use convenience maps like e2e.NativeProfiles, e2e.OCIProfiles, or e2e.AllProfiles to iterate through groups of profiles.

  8. Understand SingularityCE dependency licensing

    main

    SingularityCE uses multiple third-party dependencies managed via go.mod and go.sum.

    • Full Builds/Packages: A complete build or package of SingularityCE includes all listed dependencies.
    • Library Usage: If you import github.com/sylabs/singularity/v4 into your own Go project, you may only be subject to a subset of these dependencies depending on which parts of the package you use.
  9. Configure 'self' bootstrap with 'From' and 'Exclude' options

    main

    When performing a self bootstrap, you can refine the image composition using the following options:

    1. From:: While optional for self bootstraps, you can explicitly specify the root using From: /.
    2. Exclude:: Highly recommended to prevent unnecessary or sensitive data from being included in the image tarball. You can provide space-separated paths to exclude.

    Example definition snippet to exclude Docker data and specific user/application directories:

    Bootstrap: self
    Exclude: /var/lib/docker /home/vanessa /opt/*
  10. Getting started with SingularityCE installation and usage

    main

    Depending on your role, use the following resources to get started with SingularityCE:

    For Users

    To learn how to run and build containers, refer to the User Guide.

    For System Administrators

    To learn how to configure SingularityCE and understand its architecture and security features, refer to the Administrator Guide.

    Installation

  11. Where to ask questions and report issues in SingularityCE

    main

    SingularityCE provides several channels depending on the nature of your inquiry:

    For Usage Questions and Discussions

    Use these channels for general questions, feature ideas, or roadmap input. These provide a searchable, permanent archive:

    • GitHub Discussions: The primary place for usage questions, feature suggestions, and community announcements.
    • Google Group / Mailing List: A low-volume mailing list for announcements and discussions.
    • Slack: For real-time chat (note: history retention is limited).

    For Bug Reports and Feature Requests

    Use GitHub Issues for technical problems and formal requests. Ensure you use the correct repository:

    For Development Updates

  12. Clone the SingularityCE repository

    main

    Clone the repository using git. It is recommended to use --recurse-submodules to ensure all necessary submodules are included.

    git clone --recurse-submodules https://github.com/sylabs/singularity.git
    cd singularity

    If you have already cloned the repository without submodules, run:

    git submodule update --init

    To build a specific version, check out the corresponding release tag (e.g., v4.5.0):

    git checkout --recurse-submodules v4.5.0