stargz-snapshotter

Stargz Snapshotter is a containerd snapshotter implementation designed for lazy pulling. Instead of waiting for a full image download to complete before starting a container, it allows containers to run by fetching necessary image chunks on-demand.

eStargz is the lazily-pullable image format used by this project. Key characteristics include:

• OCI/Docker Compatibility: eStargz images are compatible with standard OCI/Docker specifications and can be pushed to standard registries (e.g., ghcr.io).
• Runtime Compatibility: They remain runnable on eStargz-agnostic runtimes like standard Docker.
• Optimization: Unlike basic lazy pulling (which can slow down file access), optimized eStargz images allow the snapshotter to prefetch and cache likely accessed files during container execution to mitigate performance drawbacks.

eStargz is a backward-compatible extension to OCI and Docker gzip image layers designed for lazy pulling.

In standard container images, the entire layer must be extracted before files can be accessed. eStargz solves this by allowing runtimes to fetch and extract only the necessary files (or specific chunks of large files) on-demand during container execution. This significantly reduces cold start times.

Key features include:

• On-demand fetching: Runtimes use the Table of Contents (TOC) to find file offsets and fetch them via HTTP Range Requests.
• Prefetching: Supports pre-loading files to mitigate runtime performance latency.
• Backward compatibility: eStargz-formatted images can be pushed to standard registries and run on eStargz-agnostic runtimes (which will simply treat them as standard gzip layers).

The project provides two distinct plugins to enable lazy pulling of eStargz images depending on your container runtime:

1. Stargz Snapshotter: A plugin for containerd (v1.4.2+) that implements the remote snapshotter interface. It communicates with containerd via gRPC over a Unix socket to provide remotely-mounted eStargz layers.

2. Stargz Store: A plugin for CRI-O or Podman that implements the additional layer store interface. It provides remotely-mounted eStargz layers through a specific filesystem structure.

Stargz Store Filesystem Structure

CRI-O/Podman access eStargz layers via a mounted filesystem at <mountpoint>/base64(imageref)/<layerdigest>/ containing:

• diff: The extracted eStargz layer.
• info: JSON-formatted metadata for the layer.
• use: Files used to notify the system of layer usage (for Garbage Collection).

Stargz snapshotter uses FUSE to mount eStargz layers. At the root of the container filesystem, a hidden state directory /.stargz-snapshotter is created for status monitoring. This directory is hidden from getdents(2) (so ls -a won't show it), but you can access it directly by path.

Each JSON file in this directory corresponds to a layer and contains:

• digest: The layer digest (matches the image manifest).
• size: Total size of the layer in bytes.
• fetchedSize: Number of bytes currently fetched.
• fetchedPercent: Percentage of the layer downloaded. When this reaches 100, the layer is fully local and no further network access is required.

Note: The directory layout and JSON structure are subject to change.

eStargz uses specific landmark files to manage the boundary between prioritized and non-prioritized files within the archive:

• .prefetch.landmark: A mandatory regular file entry placed at the boundary between the prioritized files group and the non-prioritized files group. It must have 4 bits of content 0xf.
• .no.prefetch.landmark: A mandatory regular file entry that MUST be contained in the archive if no prioritized files are present.

Both landmarks must be recorded in the TOC as a TOCEntry.

eStargz ensures the integrity of metadata (TOC) and file contents (chunks) using digests. Verification relies on the following metadata being present in the verified manifest:

1. TOC Verification: The layer descriptor referencing the eStargz layer must include the annotation containerd.io/snapshot/stargz/toc.digest. This value is the digest of the TOC. The snapshotter recalculates the TOC digest and compares it to this annotation.
2. Content Verification: Each TOCEntry for a regular file (reg) or a chunk must contain a chunkDigest property. This property holds the digest of the content of that specific entry. The snapshotter recalculates the digest of fetched chunks and verifies them against the chunkDigest recorded in the verified TOC.

An eStargz-formatted blob is a gzip-compressed tar archive containing a metadata component called the TOC (Table of Contents).

To enable seekable access, the structure follows these rules:

• Each non-empty regular file and each metadata component (TOC, Footer) MUST be separately compressed as gzip.
• Gzip headers MUST be located at the top of the blob, the top of the TOC tar header, the top of the footer, and the top of the payload of each non-empty regular file entry (except the TOC).
• Chunking: Large regular files may be split into multiple smaller gzip members called chunks. In this case, gzip headers may be located at arbitrary locations within the payload of the regular file entry.
• An eStargz blob is a concatenation of these gzip members, remaining a valid gzip blob.

After configuring storage.conf, you must start the stargz-store process within the namespace managed by Podman using the podman unshare command.

Run the following command to start the store in the background, ensuring you provide the correct paths for the data and store directories:

podman unshare stargz-store --root $HOME/.local/share/stargz-store/data $HOME/.local/share/stargz-store/store &