controller-runtime

To build controllers that interact with multiple Kubernetes clusters, use the cluster.Cluster interface to manage cluster-specific dependencies (like Client, Cache, and Scheme) separately from the main manager.Manager. The manager.Manager now embeds the Cluster interface, allowing you to add additional clusters to a single manager.

Key steps for multi-cluster setup:

1. Create a primary manager for the main cluster.
2. Create additional cluster.Cluster instances for secondary clusters.
3. Add the secondary clusters to the manager using mgr.Add(clusterInstance).
4. Use the specific cluster's methods (e.g., cluster.GetClient()) within your reconciler to perform actions on that specific cluster.

// Example: Reconciler using two different clusters
type secretMirrorReconciler struct {
	referenceClusterClient, mirrorClusterClient client.Client
}

func NewSecretMirrorReconciler(mgr manager.Manager, mirrorCluster cluster.Cluster) error {
	return ctrl.NewControllerManagedBy(mgr).
		// Watch Secrets in the reference cluster (the one the manager is tied to)
		For(&corev1.Secret{}).
		// Watch Secrets in the mirror cluster using its specific cache
		Watches(
			source.NewKindWithCache(&corev1.Secret{}, mirrorCluster.GetCache()),
			&handler.EnqueueRequestForObject{},
		).
		Complete(&secretMirrorReconciler{
			referenceClusterClient: mgr.GetClient(),
				mirrorClusterClient:    mirrorCluster.GetClient(),
		})
}

func main() {
	// 1. Setup primary manager
	mgr, err := manager.New(cfg1, manager.Options{})
	
	// 2. Setup secondary cluster
	mirrorCluster, err := cluster.New(cfg2)
	
	// 3. Add secondary cluster to manager
	if err := mgr.Add(mirrorCluster); err != nil {
		panic(err)
	}

	// 4. Initialize reconciler with both clients
	if err := NewSecretMirrorReconciler(mgr, mirrorCluster); err != nil {
		panic(err)
	}

	// 5. Start everything
	if err := mgr.Start(signals.SetupSignalHandler()); err != nil {
		panic(err)
	}
}

Because controllers often read from a cache, the data might be slightly out of date. To handle this, use one of the following strategies:

1. Optimistic Locking with Deterministic Names: Use deterministic names for objects you create. This allows the Kubernetes API server to return an error if the object already exists, preventing duplicate creation. (e.g., appending a specific number or hash to a name).
2. Action Tracking: If you cannot use deterministic names (e.g., when using generateName), track the actions you have taken and assume they need to be repeated if they do not occur within a certain timeframe (using a requeue result).
3. Full State Enforcement: Write your controller with the assumption that information will eventually be correct. Ensure the Reconcile function enforces the entire desired state of the world every time it runs.

Note: Constructing a client that reads directly from the API server instead of the cache is a last resort and generally discouraged.

To reduce recovery time during leader failover for controllers managing large caches, you can enable 'Warm Replicas'. This allows the manager to start event sources and populate caches while the instance is still on standby (before acquiring the leader lock).

When EnableWarmup is true and NeedLeaderElection is true, the controller-runtime registers the controller as a warmup runnable. The Warmup phase launches event sources and cache sync logic but does not start worker goroutines. Once leader election is won, the controller's Start method simply spins up workers against the already-populated queues.

Note: Enabling warmup on a controller that does not use leader election is a no-op.

// Enable warmup for all controllers in the manager
mgr, err := ctrl.NewManager(cfg, ctrl.Options{
	Controller: config.Controller{
		EnableWarmup: new(true),
	},
})

// ...

// Or override/enable warmup for a specific controller
builder.ControllerManagedBy(mgr).
	Named("slow-source").
	WithOptions(controller.Options{
		EnableWarmup: new(true),
	}).
	For(&examplev1.Example{}).
	Complete(reconciler)

You can initialize a ctrl.Manager using configuration loaded from a ConfigMap via the NewFromComponentConfig function. This allows for dynamic configuration of the manager without rebuilding the controller binary.

Using the default configuration type

1. Mount a ConfigMap containing the configuration into your controller pod.
2. Initialize the manager using NewFromComponentConfig by providing the name of the ConfigMap and specifying the DefaultControllerManagerConfiguration type.
3. Build your custom controller as usual.

Using a custom configuration type

1. Implement a custom ComponentConfig type.
2. Embed the ControllerManagerConfiguration type within your custom type.
3. Mount the ConfigMap containing your custom configuration into the controller pod.
4. Initialize the manager using NewFromComponentConfig by providing the name of the ConfigMap and your custom ComponentConfig type.
5. Build your custom controller as usual.