KubeSolo Documentation

KubeSolo is an ultra-lightweight, OCI-compliant, single-node Kubernetes distribution designed for edge devices, IoT gateways, and developer laptops. It aims to replace Docker or Podman by providing a full Kubernetes API and ecosystem compatibility with a RAM footprint under 200 MB.

Key characteristics:

• Single-node optimized: Removes clustering machinery (etcd quorum, leader election, multi-node networking) to reduce overhead.
• Lightweight stack: Uses containerd & crun for runtime, CoreDNS for DNS, and Kine with SQLite for storage (replacing etcd).
• Single process: The API server, controller manager, and kubelet run together in a single process.
• Scheduling: Uses a lightweight custom webhook called NodeSetter instead of the full Kubernetes Scheduler.

In container mode, you can run multiple independent KubeSolo instances side-by-side using the --name flag. Each instance will have its own:

• Container (kubesolo-<name>)
• Data volume
• Kubeconfig context
• Docker context (if using d2k)

Always pass the same --name to management commands like upgrade, reset, uninstall, kubeconfig fetch, and d2k fetch to target the specific instance.

Because the KubeSolo node runs in its own network namespace, services using NodePort, LoadBalancer, or pods using hostPort are not automatically reachable from the host.

You must explicitly publish these ports when creating the container (similar to Kind's extraPortMappings).

• Using kubesoloctl: Use the --container-ports flag.
• Using raw docker run: Add the required -p host_port:container_port mappings during the docker run command.

KubeSolo uses a non-standard directory for CNI plugin binaries. If you install an external CNI (like Cilium), you must ensure the plugin binaries are placed in KubeSolo's specific path, otherwise pod sandbox creation will fail with a 'failed to find plugin' error.

Required Path Mappings:

• Plugin binaries (bin_dirs): /var/lib/kubesolo/containerd/cni/plugins (Note: Stock Kubernetes defaults to /opt/cni/bin)
• CNI config (conf_dir): /etc/cni/net.d (Matches stock default; no override needed)

KubeSolo does not include a kube-scheduler. Instead, a mutating webhook assigns pods to the single node by patching spec.nodeName directly.

Implications for CNI and Workloads:

• Replica Counts: Because there is no scheduler to spread workloads, any component that defaults to multiple replicas (like cilium-operator) will attempt to run all replicas on the same node.
• Host Port Conflicts: If multiple replicas attempt to bind to the same host port on the single node, the kubelet will reject them with a NodePorts status.
• Best Practice: Always set the replica count to 1 for CNI components and other workloads that rely on host ports or anti-affinity.

kubesoloctl operates in different modes depending on your environment and use case:

• service (default on Linux): Installs the KubeSolo binary as a system service (systemd, OpenRC, etc.). This is the production path for edge devices and gateways.
• container: Runs KubeSolo inside a container. This is the required mode for macOS and is recommended for development/CI on Windows (WSL2) or Linux. It requires a running container engine (Docker Engine / Docker Desktop).
• daemon: Runs KubeSolo as a background process without an init system. Best for minimal hosts.
• foreground: Keeps the process in the terminal, useful for debugging.

To install KubeSolo directly on a Linux host as a system service, use the install command with sudo. This performs pre-flight checks, installs the binary to /usr/local/bin/kubesolo, configures the service, and merges the admin kubeconfig into your ~/.kube/config.

sudo kubesoloctl install

# Verify installation
kubectl get nodes --watch    # wait until STATUS: Ready
kubectl get pods -A

To run KubeSolo inside a container (ideal for local development), use the --run-mode=container flag. On macOS, this mode is applied automatically. KubeSolo will publish its API server on a random localhost port and automatically merge/point your kubeconfig to it.

Note: If the host port changes (e.g., after an upgrade or reset), you must refresh your kubeconfig using kubesoloctl kubeconfig fetch.

# On macOS this is implied; elsewhere request it explicitly:
kubesoloctl install --run-mode=container

# Verify installation
kubectl get nodes --watch

# If the host port changes, refresh kubeconfig:
kubesoloctl kubeconfig fetch

After installation or upgrade, verify that the Cilium agent and operator are running correctly and that the binary is in the expected KubeSolo directory.

Verification Steps:

1. Check Cilium status: cilium status --wait
2. Check pod status: kubectl -n kube-system get pods -l k8s-app=cilium
3. Confirm binary location: ls -l /var/lib/kubesolo/containerd/cni/plugins/cilium-cni

Cleanup: If failed operator pods are stuck in the system, delete them using:

kubectl -n kube-system delete pod -l io.cilium/app=operator --field-selector status.phase=Failed