firecracker-containerd

firecracker-containerd enables the use of containerd to manage Firecracker microVMs. It provides the isolation of a KVM hypervisor while maintaining the fast startup and low overhead characteristics of containers. It uses OCI image formats to maintain compatibility with the container ecosystem.

Key components include:

• Control Plugin (firecracker-control): Manages the microVM lifecycle via a specific control API. This is compiled into a specialized containerd binary.
• Runtime (runtime): An out-of-process shim runtime that links containerd (on the host) to the Firecracker VMM using ttrpc.
• Agent (agent): Runs inside the microVM and uses runC via containerd-shim-runc-v1 to create standard Linux containers within the microVM.
• Root Filesystem Image Builder (tools/image-builder): A tool to construct microVM root filesystems containing runc and the firecracker-containerd agent.

To work with firecracker-containerd, you must understand the distinction between Host and Guest shims:

• Host Shim (FC Runtime): A process running on the host machine that manages interactions with one or more microVMs. It handles request forwarding to the microVM and manages I/O streams coming out of the microVM.
• Guest Shim (Internal FC Agent): A process running inside the microVM that manages activities inside the VM, such as spawning containers based on requests from the Host Shim and forwarding I/O streams from the VM to the Host Shim.
• FC-Control: A custom containerd GRPC plugin used to handle microVM-specific APIs that lack existing containerd equivalents (e.g., StartVM).
• vm_id: A unique identifier for each Firecracker microVM. It is used to reference which VM a specific container is intended to run in (typically mapping 1-to-1 with an ECS task or EKS Pod).

Firecracker-containerd supports two primary models for connecting a VM's tap device to a host network. Choosing the right model depends on whether you want the VM to act as a separate entity on the network or transparently inherit the host's network identity.

1. Traffic Control (TC) Redirect (Preferred)

This model uses the Linux Kernel's Traffic Control system (specifically u32 filters) to redirect packets between the VM's tap device and another device (like a veth pair) in the host's network namespace.

Key Characteristics:

• Transparency: The VM guest operates as if its NIC is the same as the host's device, inheriting the same MAC address, IP, and routes.
• CNI Integration: Highly compatible with CNI plugins because the VM does not need its own unique IP on the network; it reuses the identity of the device it is redirecting to.
• Pros: Minimal requirements on the host network; supports IPv6.
• Cons: Requires maintaining code for the relatively undocumented TC subsystem; host-side processes in the same namespace might face connectivity issues because the IP is bound to the redirecting device.

2. Network Bridge

This model attaches the VM's tap device to a virtual network bridge, acting like a software switch.

Key Characteristics:

• Identity: The VM and its tap device have their own unique MAC addresses and are treated as separate entities on the network.
• Pros: Uses well-documented standard Linux bridging patterns.
• Cons: The VM must be assigned its own unique IP address on the network, which adds complexity to network management and configuration.

firecracker-containerd enables running containers with Firecracker VM isolation by integrating with containerd. It implements the V2 runtime interface and introduces a custom "control" API to manage the lifecycle of a Firecracker microVM that hosts multiple containers.

The architecture consists of three primary components:

1. Control Plugin: Manages the runtime lifecycle and implements the control API by proxying commands to the runtime. This plugin is compiled into a specialized containerd binary.
2. Runtime (Shim): An out-of-process shim communicating via ttrpc. It bridges containerd (host side) to the Firecracker VMM (for VM lifecycle) and to the Agent (inside the microVM for container lifecycle).
3. Agent: Runs inside the microVM. It executes control instructions from the runtime, emits metrics/events, and proxies STDIO. It uses runC (via containerd-shim-runc-v1) to create standard Linux containers within the VM.

In firecracker-containerd, remote snapshotters (like stargz-snapshotter or nydus-snapshotter) are run inside the microVM rather than on the host. This bypasses Firecracker's lack of virtio-fs support by allowing the lazy-loaded filesystem to exist natively within the guest.

Key architectural components include:

• demux-snapshotter: A host-side proxy that receives requests from containerd and routes them to the specific remote snapshotter running inside a microVM.
• firecracker-containerd agent: The guest-side daemon managing containers and implementing the Task API.
• socat: Used inside the guest to expose the remote snapshotter's Unix socket over vsock, allowing the host-side demux-snapshotter to communicate with it.
• remote snapshotter: The component inside the guest responsible for lazy loading image content.

Note: Unlike standard devmapper workflows, the microVM is launched before preparing container snapshots when using remote snapshotters.

To set up a firecracker-containerd environment on Debian Bullseye, you must install Go 1.23, Docker CE, device-mapper, and several build tools. Note that the Go 1.23 package installs the binary under /usr/lib/go-1.23/bin, which must be added to your PATH.

#!/bin/bash

cd ~

# Install git, Go 1.23, make, curl
sudo mkdir -p /etc/apt/sources.list.d
echo "deb http://ftp.debian.org/debian bullseye-backports main" | \
  sudo tee /etc/apt/sources.list.d/bullseye-backports.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
  install --yes \
  golang-1.23 \
  make \
  git \
  curl \
  e2fsprogs \
  util-linux \
  bc \
  gnupg

# Debian's Go 1.23 package installs "go" command under /usr/lib/go-1.23/bin
export PATH=/usr/lib/go-1.23/bin:$PATH

cd ~

# Install Docker CE
# Docker CE includes containerd, but we need a separate containerd binary, built
# in a later step
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
apt-key finger docker@docker.com | grep '9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88' || echo '**Cannot find Docker key**'
echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get \
     install --yes \
     docker-ce aufs-tools-
sudo usermod -aG docker $(whoami)

# Install device-mapper
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y dmsetup