Kilo Documentation

Kilo is a multi-cloud network overlay built on WireGuard for Kubernetes. It provides an encrypted Layer 3 network that can span across different data centers and public clouds.

Key capabilities include:

• Fully connected Pod networks: Enables communication even between nodes in different networks or behind NAT.
• Multi-cloud clusters: Allows pools of nodes in different locations to communicate securely.
• VPN access: Enables external clients to VPN into a cluster to access services or debug resources.
• Multi-cluster services: Enables services to span across different Kubernetes clusters.

The kg agent is the core component of a Kilo mesh that runs on every Kubernetes node. Its primary responsibilities include:

• Mesh Membership: Adding the node to the Kilo mesh.
• CNI Management: Installing and managing CNI (Container Network Interface) configurations on the node.
• WireGuard Setup: Configuring the WireGuard network interface.
• Network Maintenance: Maintaining routing table entries and iptables rules to ensure connectivity between locations.

Kilo can connect independent Kubernetes clusters over a secure WireGuard connection, allowing clusters to provide private services to one another. This enables workloads in one cluster (e.g., an on-prem cluster) to consume services in another (e.g., an AWS cluster with GPUs) without exposing them via Ingress or NodePort.

Critical Requirement: To ensure full routability, the declared IP ranges must be non-overlapping. This includes the Kilo, pod, and service CIDRs for all connected clusters.

Kilo uses WireGuard to create a mesh between nodes in a cluster. The Kilo agent, kg, runs on every node to manage public/private keys and routing rules.

Kilo can be used in two modes:

1. Independent networking provider: Acts as the primary CNI.
2. Add-on mode: Complements an existing CNI (like Flannel) by handling the network between different locations/clouds, while the existing CNI handles local networking within those locations.

Kilo uses a Kubernetes ValidatingAdmissionWebhook to prevent faulty Peer configurations from being applied to the cluster.

When a ValidatingWebhookConfiguration is active, the Kubernetes API server sends an AdmissionReviewRequest to the Kilo Peer Validation service whenever a CREATE or UPDATE operation is attempted on a Peer resource.

• If the webhook returns "allowed": true, the operation proceeds.
• If the webhook returns "allowed": false, the operation is denied.
• The default failurePolicy is set to Fail, meaning if the webhook service is unavailable, the API server will deny the changes to ensure safety.

Kilo determines how to group nodes into logical locations using the following priority:

1. The topology.kubernetes.io/region node label.
2. A custom label specified via the --topology-label=<label> command line flag.
3. The kilo.squat.ai/location node annotation.

If no custom label or annotation is provided, Kilo uses the Kubernetes region label. To manually group nodes into a specific location (e.g., gcp), use the kilo.squat.ai/location annotation.

Before building or testing Kilo, ensure you have the following utilities installed on your system:

• go (used for code formatting and running unit tests)
• make
• jq
• git
• curl
• docker (required for binary compilation and container builds)

Use make commands to execute different levels of testing and code quality checks:

• make unit: Runs unit tests.
• make lint: Lints the repository code.
• make e2e: Runs basic end-to-end tests (note: these may be flaky).
• make test: Runs all tests with a single command.

To monitor the Kilo DaemonSet using the kube-prometheus stack, apply the provided PodMonitor manifest to your cluster.

kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/podmonitor.yaml

Generate the static production content by running yarn build. The resulting static files will be located in the build directory and can be hosted on any static content hosting service.

$ yarn build