DOCSEARCHbyAlgolia
Submit Docs

Sveltos Add-on Controller Documentation

repository·main·Indexed 19 days ago

https://github.com/projectsveltos/addon-controller

Sveltos is a Kubernetes add-on controller for centralized management of applications and configurations across multiple clusters from a single management cluster. It supports Helm charts, raw YAML, Kustomize, Carvel ytt, and Jsonnet. Key features include multi-format support, ordered deployment, event-driven deployments, and a Pull Mode for air-gapped environments. It utilizes ClusterProfiles for global consistency and Profiles for tenant-specific granular control, offering various sync modes including OneTime, Continuous, and ContinuousWithDriftDetection.

Tokens
3.6K
Snippets
14
Records
18
Agent score
68%

What's inside Sveltos addon-controller

  1. What is Sveltos?

    main

    Sveltos is a Kubernetes add-on controller designed to simplify the deployment and management of applications and add-ons across a fleet of clusters. It operates from a management cluster and can programmatically manage add-ons on any registered cluster (including itself).

    Key capabilities include:

    • Multi-format support: Manages Helm charts, raw YAML, Kustomize, Carvel ytt, and Jsonnet.
    • Templating: Uses templates to allow reusable add-on definitions with minor variations (e.g., different configuration values) per cluster.
    • Ordered Deployment: Provides precise control over deployment order within a Profile/ClusterProfile and supports dependencies between profiles.
    • Event-driven: Supports dynamic deployments in response to specific cluster events.
    • Pull Mode: Supports air-gapped or edge environments where managed clusters initiate outbound connections to the management cluster, eliminating the need for inbound firewall rules or VPNs.

  2. How Sveltos add-on deployment works

    main

    Deployment is driven by selecting clusters and defining the required add-ons.

    1. Cluster Selection: The management cluster selects target clusters using a Kubernetes label selector.
    2. Add-on Definition: You specify which add-ons to deploy. Supported add-ons include:
      • Helm releases
      • Kubernetes resource YAMLs
      • Kustomize resources

    As soon as a cluster matches the clusterSelector in a ClusterProfile, Sveltos automatically deploys the referenced features.

  3. Profiles vs. ClusterProfiles

    main

    Sveltos uses two primary abstractions to manage configurations, distinguished by their scope and intended administrator role:

    1. ClusterProfiles:

      • Scope: Applied across all clusters in any namespace.
      • Use Case: Ideal for platform administrators to maintain global consistency for settings like networking, security, and resource allocation.

    2. Profiles:

      • Scope: Limited to a specific namespace.
      • Use Case: Designed for tenant administrators to provide granular control. This allows different teams to manage their own sets of clusters from the management cluster without impacting other tenants.

  4. Quick Start Sveltos locally with make

    main

    The fastest way to test Sveltos is using the provided make quickstart command. This command:

    1. Creates a kind cluster with Sveltos and ClusterAPI installed.
    2. Provisions a workload cluster using Docker as the infrastructure provider.

    This allows you to experience the full fleet management flow locally in minutes.

    make quickstart

  5. How health validation works in Sveltos

    main

    Sveltos performs health validation on managed resources using three primary mechanisms. When a ValidateHealth policy is triggered, the controller evaluates the health of resources (or metrics) based on the following hierarchy:

    1. Job Checks: If a JobCheck is defined, it uses a specialized implementation (provided by Sveltos Enterprise) to validate job-related health.
    2. Lua Scripts: A Lua script can be provided to perform complex logic. The script has access to:
      • A global metrics table containing scalar values from MetricQueries (e.g., metrics["cpu_usage"]).
      • A global obj table representing the Kubernetes resource being inspected (if a Kind is specified).
      • The script must define an evaluate(obj) function that returns a table containing healthy (boolean) and message (string).
    3. CEL Rules: Common Expression Language (CEL) rules can be used to perform declarative checks against the resource's fields.

    If a Kind is not specified in the policy, Sveltos performs a metrics-only check, executing the Lua script once using only the collected metric data without a resource object.

    -- Example Lua script for a ValidateHealth policy
function evaluate(obj)
  -- Accessing a metric collected via MetricQueries
  local cpu_load = metrics["cpu_usage"]
  
  -- Accessing resource fields via the 'obj' table
  local replicas = obj.spec.replicas

  if cpu_load > 0.8 and replicas < 3 then
    return { healthy = false, message = "CPU load too high for current replica count" }
  end

  return { healthy = true }
end

  6. Run the addon-controller CLI

    main

    The addon-controller is executed as a single binary. The entrypoint calls app.Run(), which initializes and starts the controller's main execution loop, including command parsing and service orchestration. To use the controller, run the compiled binary directly.

    ./addon-controller

  7. Deploy Helm charts and resources using ClusterProfile

    main

    You can use a ClusterProfile to deploy Helm charts and reference existing Kubernetes resources (like Secrets or ConfigMaps) from the management cluster to matching clusters.

    In the example below, any cluster with the label env: prod will receive the Kyverno Helm chart and the resources contained in the specified Secret and ConfigMap.

    apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: deploy-kyverno
spec:
  clusterSelector:
    matchLabels:
      env: prod
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v3.8.1
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
    values: |
      admissionController:
        replicas: 3
  policyRefs:
  - name: storage-class
    namespace: default
    kind: Secret
  - name: contour-gateway
    namespace: default
    kind: ConfigMap

  8. Deploy Kustomize resources using ClusterProfile

    main

    A ClusterProfile can also reference Kustomize resources. In this pattern, Sveltos uses a kustomizationRefs to point to a resource (like a Flux GitRepository) that contains the desired manifests.

    apiVersion: config.projectsveltos.io/v1beta1
kind: ClusterProfile
metadata:
  name: flux-system
spec:
  clusterSelector:
    matchLabels:
      env: fv
  syncMode: Continuous
  kustomizationRefs:
  - namespace: flux-system
    name: flux-system
    kind: GitRepository
    path: ./helloWorld/
    targetNamespace: eng

  9. Configure SyncMode for add-on deployments

    main

    The syncMode field in a Profile or ClusterProfile determines how Sveltos manages the lifecycle and consistency of the add-ons:

    • OneTime: Used for bootstrapping. It performs a one-shot configuration injection (e.g., installing CNI plugins or package managers) and then hands over management to the workload cluster's own tools.
    • Continuous: Continuously monitors for changes in the Profile/ClusterProfile and automatically applies them to matching clusters to ensure centralized consistency.
    • ContinuousWithDriftDetection: Continuously monitors and automatically corrects configuration drifts. If the actual state in a managed cluster deviates from the desired state defined in the management cluster, Sveltos will reconcile it.

  10. Register default indexes with AddDefaultIndexes

    main

    Use AddDefaultIndexes to register the standard set of indexes within a controller manager. This function automatically initializes the ByClusterNamespace and ByClusterName indexes. It requires a context.Context and a sigs.k8s.io/controller-runtime Manager.

    import (
	"context"
	"sigs.k8s.io/controller-runtime"
	"github.com/projectsveltos/addon-controller/api/v1beta1/index"
)

func Setup(mgr ctrl.Manager) error {
	return index.AddDefaultIndexes(context.Background(), mgr)
}

  11. Calculate the ClusterReport name

    main

    The GetClusterReportName function generates the unique name for a ClusterReport resource based on the profile used, the cluster type, and the cluster name.

    If the profile kind is a Profile (configv1beta1.ProfileKind), the name is prefixed with p--. If it is a ClusterProfile, the prefix is empty. The components are joined using the -- separator.

    Name Format:

    • For Profiles: p--<profileName>--<clusterType>--<clusterName>
    • For ClusterProfiles: <profileName>--<clusterType>--<clusterName>
    // Example logic for name generation:
// profileKind: "Profile", profileName: "my-profile", clusterName: "prod-cluster", clusterType: "k8s"
// Result: "p--my-profile--k8s--prod-cluster"

// profileKind: "ClusterProfile", profileName: "global-profile", clusterName: "edge-cluster", clusterType: "k8s"
// Result: "global-profile--k8s--edge-cluster"

  12. Retrieve CRD YAML definitions via the crd package

    main

    The crd package provides functions to programmatically retrieve the raw YAML byte slices for the Custom Resource Definitions (CRDs) used by Sveltos. These functions are useful if you are building tools or extensions that need to apply or validate the Sveltos CRD schemas.

    import "github.com/projectsveltos/addon-controller/lib/crd"

// Example: Getting the ClusterProfile CRD definition
clusterProfileYaml := crd.GetClusterProfileCRDYAML()

// Example: Getting the Profile CRD definition
profileYaml := crd.GetProfileCRDYAML()