k3d Documentation

The default podman network has DNS disabled. To allow k3d cluster nodes to communicate via DNS, create a new Podman network (e.g., named k3d) where DNS is enabled.

podman network create k3d
# Verify DNS is enabled
podman network inspect k3d -f '{{ .DNSEnabled }}'

When importing images into a k3d cluster, you can choose between three modes depending on your environment and performance requirements:

1. Auto: The default mode. k3d automatically selects between direct and tools-node. It selects tools-node for remote container runtimes to reduce network overhead, and direct for local runtimes.
2. Direct: Loads images directly into the k3s nodes. This mode is efficient as it does not spawn a separate container or write intermediate files.
3. Tools Node: Spawns a k3d-tools container within the container runtime, copies the images to that runtime, and then loads them into the k3s nodes from within that container.

Follow these steps to create a single-node k3s cluster, connect to it with kubectl, and then delete it.

1. Create the cluster: k3d cluster create <CLUSTER_NAME>
2. (Optional) Merge the kubeconfig to switch context automatically: k3d kubeconfig merge <CLUSTER_NAME> --kubeconfig-switch-context
3. Use kubectl to interact with the cluster.
4. Delete the cluster: k3d cluster delete <CLUSTER_NAME>

To use a custom Docker registry container (not managed by k3d):

1. Run your registry container (e.g., registry:2).
2. Add the registry to your registries.yaml configuration.
3. Connect the registry container to the k3d network so the cluster nodes can reach it: docker network connect k3d-k3s-default <registry-container-name>.

docker volume create local_registry
docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 12345:5000 registry:2

# Connect to k3d network
docker network connect k3d-k3s-default registry.localhost

To expose services using a NodePort, you map a specific port from a specific agent node to your host.

1. Create the cluster mapping a host port to a specific agent node. For example, mapping host port 8082 to port 30080 on agent-0: k3d cluster create mycluster -p "8082:30080@agent:0" --agents 2

2. Create a NodePort service using a manifest that specifies the nodePort within the Kubernetes default range (30000-32767).

3. Access the service via curl localhost:8082/.

Warning: You can attempt to expose the entire NodePort range (e.g., -p "30000-32767:30000-32767@server:0"), but because Docker creates iptable entries and a proxy process per port-mapping, this may take a very long time or freeze your system.

You can pass flags to specific Kubernetes components (like kube-apiserver, kube-scheduler, or kubelet) using the --k3s-arg flag.

Important Syntax Rules:

• The k3s flag itself must include dashes (e.g., --kube-apiserver-arg).
• The internal Kubernetes flags (e.g., feature-gates) should not include dashes, as k3s adds them internally.
• Use the @role:* suffix to target specific node roles (e.g., @server:* or @agent:*).

k3d cluster create \
  --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*' \
  --k3s-arg '--kube-scheduler-arg=feature-gates=EphemeralContainers=true@server:*' \
  --k3s-arg '--kubelet-arg=feature-gates=EphemeralContainers=true@agent:*'

You can create a pull-through registry to mirror a remote registry (like Docker Hub) and cache images locally. This speeds up image pulls for subsequent clusters.

1. Create the registry container: Use k3d registry create with the --proxy-remote-url flag.
2. Configure K3s mirrors: Create a registry.yml file to tell K3s to use your new registry as a mirror for specific domains.
3. Create a cluster using the proxy: Use the --registry-use and --registry-config flags during cluster creation.

The docgen tool is used exclusively to generate the command tree documentation for the k3d usage commands. The output files are written to ../docs/usage/commands/.

Note: You must execute the script from within the docgen directory because the relative path to the docs/ directory is hardcoded in the tool.

# ensure that you're in the docgen dir, as the relative path to the docs/ dir is hardcoded
cd docgen

# run
./run.sh

To use k3d, you must have the following installed:

1. docker: k3d v5.x.x requires at least Docker v20.10.5 (runc >= v1.0.0-rc93).
2. kubectl: Required to interact with the Kubernetes clusters created by k3d.