macOS-vz-kubelet

macOS-vz-kubelet is a Virtual Kubelet provider that enables running native macOS virtual machines as Kubernetes pods on Apple Silicon Macs. Unlike traditional QEMU/KVM approaches, it uses Apple's Virtualization.framework to achieve near-native performance.

Key Features:

• Native Performance: Runs macOS VMs directly on Apple Silicon hardware.
• Hybrid Pods: Supports pods where the first container (index 0) is the macOS VM, and subsequent containers (1..N) are standard Docker side-cars running on the local host's Docker daemon.
• OCI-Compliant Imaging: Uses a custom, compressed OCI image format to distribute large macOS VM images via standard registries.
• Automated Networking: Provides NAT (default) or Bridged networking modes.

The kubelet acts as a Virtual Kubelet provider on the Mac host, orchestrating two main components from a single pod specification:

1. Virtualization Framework: Provisions and runs the macOS VM as the first container in the pod (index 0).
2. Custom OCI Image Format: Distributes VM images as compressed OCI artifacts. The kubelet pulls these using the oras-go library.

Lifecycle and Resource Management:

• Scheduling: CPU and memory requests in the pod spec determine the VM size.
• Lifecycle: Supports pod creation and deletion. Note that updates to the pod spec require recreating the pod.
• Side-cars: Any containers defined after the first one in the pod spec run as regular containers on the host's local Docker daemon.
• Interaction: kubectl exec and kubectl attach commands work by communicating with the VM over SSH.

The kubelet supports two networking modes:

NAT (Default)

• The VM receives a local IP via NAT.
• Best for cases where external reachability is not required.
• kubectl exec/attach works via SSH on the local IP, which the kubelet discovers from the host ARP table.

Bridged

• Provides the VM with a routable IP by attaching it to a host network interface (e.g., a tagged VLAN with DHCP).
• Each VM interface uses a generated MAC address for DHCP lease tracking.
• The kubelet discovers the VM IP by sniffing for the MAC address using libpcap.
• Configuration: Set the VZ_BRIDGE_INTERFACE environment variable to the name of the host interface.
• Requirement: Requires Apple-approved VMNet and VM Networking capabilities. You must build a release binary signed with resources/release.entitlements after obtaining approval.

To run a macOS workload, follow these steps:

1. Create a base VM image using a tool like macosvm.
2. Package and push the image using the oras-macos-vz CLI tool in the custom OCI format:

   oras-macos-vz push -h

3. Write a pod manifest that references the OCI image you just pushed.
4. Deploy the pod to your Kubernetes cluster. The kubelet will pull the image and boot the VM.
5. Interact with the VM using standard Kubernetes commands:

   kubectl exec -it <pod-name> -- <command>

# 1. Package and push
oras-macos-vz push -h

# 2. Run pod (via kubectl apply)
kubectl apply -f pod-manifest.yaml

# 3. Interact
kubectl exec -it <pod-name> -- /bin/zsh

Builds are performed on macOS using make. Tooling versions are managed via .mise.toml.

Build Commands:

make snapshot   # Local development build
make release    # Signed release build (requires RELEASE_CERTIFICATE_NAME and RELEASE_PROVISION_PROFILE_PATH)

Ad-hoc Signing: If you are building locally and only need NAT networking, you must sign the binary with the required entitlements:

codesign --entitlements resources/vz.entitlements -s - <YOUR BINARY PATH>

Note: Bridged networking requires additional vmnet entitlements found in resources/release.entitlements, which require Apple approval.

make snapshot
# or
make release

# For ad-hoc signing:
codesign --entitlements resources/vz.entitlements -s - <YOUR BINARY PATH>

The virtual-kubelet binary implements a Kubernetes node that uses Apple's Virtualization.framework to run macOS Virtual Machines as pods. It uses the Virtual Kubelet framework to bridge Kubernetes pod lifecycle management with macOS virtualization.

To run the binary, ensure you have a valid KUBECONFIG (or it will default to ~/.kube/config) and the necessary TLS certificates if authentication is enabled.

# Example execution with common flags
./virtual-kubelet \
  --nodename "my-macos-node" \
  --provider-id "macos-vz-provider-1" \
  --log-level "debug" \
  --disable-taint

The example/ directory contains ready-to-adapt Kubernetes manifests for different macOS VM scenarios:

• pod.yml: A minimal macOS VM pod.
• pod-gitlab-sidecar.yml: A hybrid pod containing a macOS VM and a GitLab Runner Docker side-car.
• deployment.yml: A Deployment managing multiple macOS VM pods.
• oci_manifest.json and config.json: Definitions for the custom OCI image format used by this project.

The kubelet automatically applies a taint to the node to prevent unschedulable pods from landing on it unless they have the appropriate tolerations. You can customize this taint using the following environment variables:

• VKUBELET_TAINT_KEY: The key for the taint. Defaults to virtual-kubelet.io/provider.
• VKUBELET_TAINT_EFFECT: The effect of the taint. Supported values: NoSchedule, NoExecute, PreferNoSchedule. Defaults to NoSchedule.
• VKUBELET_TAINT_VALUE: The value of the taint. Defaults to macos-vz.

The following environment variables control the underlying virtualization and networking behavior:

• VZ_BRIDGE_INTERFACE: The identifier for the network interface used for bridging.
• KUBELET_PORT: The port on which the kubelet listens. If set, overrides the default --listen-port logic.
• VKUBELET_POD_IP: The internal IP address of the kubelet pod.
• DOCKER_HOST: If set, the kubelet will use this host to connect to a Docker daemon for managing non-macOS containers.
• OTEL_SERVICE_NAME: The service name used for OpenTelemetry tracing.