vCluster

vCluster provides virtual Kubernetes clusters with several core capabilities:

• Isolated Virtual Control Plane: Each Tenant Cluster has its own API server, controller manager, and data store, ensuring complete Kubernetes API isolation.
• Strong Tenant Isolation: Tenants receive admin access within their Tenant Cluster while maintaining minimal permissions on the host Control Plane Cluster.
• Resource Syncing: Supports bidirectional synchronization of Kubernetes resources (pods, services, secrets, configmaps, CRDs, etc.) between the Tenant Cluster and the Control Plane.
• Sleep Mode: Allows pausing inactive Tenant Clusters to conserve resources, with instant wake-up capabilities.
• High Availability: Supports multiple replicas with leader election and can use embedded etcd or external databases like PostgreSQL, MySQL, or RDS.
• Standalone Deployment: Can be run without a Control Plane Cluster on dedicated infrastructure or bare metal.
• Integrations: Native support for cert-manager, external-secrets, KubeVirt, Istio, and metrics-server.

Note on Deployment Modes:

• Shared / Dedicated Nodes: These modes leverage the Control Plane Cluster's CNI, CSI, ingress, and other infrastructure. Resource syncing and host-cluster integrations apply here.
• Private Nodes / Standalone: These modes bring their own CNI, CSI, and platform components.

vCluster supports four primary architectures, offering different levels of isolation and infrastructure requirements:

Configuration Examples

Shared Nodes

Tenant Clusters share the Control Plane Cluster's nodes. Workloads run as regular pods in a namespace.

sync:
  fromHost:
    nodes:
      enabled: false  # Uses pseudo nodes

Dedicated Nodes

Tenant Clusters get their own set of labeled nodes on the Control Plane Cluster.

sync:
  fromHost:
    nodes:
      enabled: true
      selector:
        labels:
          tenant: my-tenant

Private Nodes (v0.27+)

External nodes join the Tenant Cluster directly with their own CNI, CSI, and networking stack.

privateNodes:
  enabled: true
controlPlane:
  service:
    spec:
      type: NodePort

vCluster Standalone (v0.29+)

Run vCluster without any Control Plane Cluster. The Virtual Control Plane is deployed directly on bare metal or VMs.

controlPlane:
  standalone:
    enabled: true
    joinNode:
      enabled: true
privateNodes:
  enabled: true

Auto Nodes (v0.28+)

Automatically provision and deprovision private nodes based on workload demand using providers like Karpenter.

autoNodes:
  enabled: true
  nodeProvider: <provider>
privateNodes:
  enabled: true

The test suite is organized into two distinct layers to allow for flexible testing across different vCluster configurations:

1. Spec functions (test_*/): These are self-describing logic blocks. They contain Describe text and feature labels (e.g., labels.Core, labels.Pods) but do not bind to a specific cluster. This allows the same spec to be reused across multiple suites.
2. Suite files (suite_*_test.go): These files define a specific vCluster configuration. A suite owns:
   • An embedded vcluster-*.yaml template.
   • The vCluster name.
   • The vCluster lifecycle (using Ordered Ginkgo suites and BeforeAll to provision the cluster).
   • Scheduling labels (e.g., labels.PR for gating, or a primary label like labels.Rootless).

The Lifecycle Helper: Suites use setup/lazyvcluster (a wrapper around vcluster.Create) to provision clusters. This ensures that vClusters are created lazily in the suite's BeforeAll and destroyed in AfterAll. If a spec fails, the framework keeps the vCluster alive and attaches diagnostics (rendered config, pods, events, syncer logs) to the report.

To verify vCluster conformance against Kubernetes v1.21, you must set up a host cluster (GKE is recommended), deploy a specifically configured vCluster, and execute tests using Sonobuoy.

Prerequisites

• A GKE cluster with at least 2 nodes.
• The vcluster CLI (version v0.4.5 or newer).
• Sonobuoy installed.

Step 1: Prepare the Host Cluster (GKE)

Create a GKE cluster and ensure a firewall rule is in place to allow incoming connections or NodePort traffic, otherwise tests will fail.

Step 2: Deploy the vCluster

You must use a custom values.yaml to configure the vCluster for conformance testing. This configuration uses a specific k3s image, applies broad tolerations to handle node tainting during tests, and configures the syncer to sync nodes and priority classes while disabling ingress sync.

Step 3: Execute Sonobuoy Tests

Run the sonobuoy run command in certified-conformance mode using the specified conformance and image versions.

### 1. Create GKE cluster
export PROJECT_NAME=my-google-cloud-project
export CLUSTER_NAME=conformance-test
export CLUSTER_ZONE=europe-west3-a
export CLUSTER_VERSION=1.21.5-gke.1302
export CLUSTER_CHANNEL=regular

gcloud beta container --project $PROJECT_NAME clusters create $CLUSTER_NAME \
   --zone $CLUSTER_ZONE --no-enable-basic-auth --cluster-version $CLUSTER_VERSION \
   --release-channel $CLUSTER_CHANNEL --enable-ip-alias --no-enable-master-authorized-networks \
   --addons GcePersistentDiskCsiDriver --node-locations $CLUSTER_ZONE
   
gcloud compute firewall-rules --project $PROJECT_NAME create conformance-firewall-rules --direction=INGRESS --network=default --action=ALLOW --rules=tcp --source-ranges=0.0.0.0/0 --description="vcluster conformance test firewall rule"

### 2. Create the vcluster
# values.yaml content:
# vcluster:
#   image: rancher/k3s:v1.21.5-k3s1
# tolerations:
# - operator: "Exists"
# rbac:
#   clusterRole:
#     create: true
# syncer:
#   extraArgs:
#   - --sync-all-nodes
#   - --sync-node-changes
#   - --fake-nodes=false
#   - --fake-kubelets=false
#   - --enable-priority-classes
#   - --disable-sync-resources=ingresses

vcluster create vcluster -n vcluster -f values.yaml --expose
vcluster connect vcluster -n vcluster

### 3. Run Tests
export KUBECONFIG=./kubeconfig.yaml
export CONFORMANCE_VERSION=v1.21.1
export SONOBUOY_IMAGE_VERSION=v0.52.0
export SONOBUOY_LOGS_IMAGE_VERSION=v0.3

sonobuoy run \
  --mode=certified-conformance \
  --kube-conformance-image-version=$CONFORMANCE_VERSION \
  --sonobuoy-image=sonobuoy/sonobuoy:$SONOBUOY_IMAGE_VERSION \
  --systemd-logs-image=sonobuoy/systemd-logs:$SONOBUOY_LOGS_IMAGE_VERSION \
  --wait