DOCSEARCHbyAlgolia
Submit Docs

kube-fledged

repository·master·Indexed 23 days ago

https://github.com/senthilrch/kube-fledged

A Kubernetes operator that manages container image caches on worker nodes to enable near-instant pod startup times. It allows users to pre-pull a defined list of images onto specific nodes via the ImageCache Custom Resource, which is particularly useful for rapid scaling, serverless functions, and IoT/Edge computing.

Tokens
8.9K
Snippets
14
Records
28
Agent score
78%

What's inside kube-fledged

  1. What is kube-fledged?

    master
    kube-fledged is a Kubernetes operator designed to create and manage a cache of container images directly on the worker nodes of a cluster. By pre-pulling a defined list of images onto specific worker nodes, it ensures that application pods can start almost instantly without waiting for images to be pulled from a remote registry. This is particularly useful for rapid scaling, serverless functions, IoT/Edge computing with intermittent connectivity, and managing access to private registries.

  2. How kube-fledged works with Kubernetes Custom Resources

    master

    kube-fledged extends the Kubernetes API by defining a Custom Resource of kind ImageCache. The core logic is managed by a custom controller named kubefledged-controller.

    Users interact with the system by using standard kubectl commands to create or delete ImageCache resources. The kubefledged-controller manages the image lifecycle using the following mechanisms:

    • Image Manager Routine: Responsible for pulling and deleting images using Kubernetes Jobs.
    • Refresh Worker: If enabled, periodically refreshes the image cache to ensure it remains up to date.
    • Status Updates: The controller automatically updates the status field of the ImageCache resource to reflect the current state of image pulls, refreshes, and deletions.

  3. How the Cluster Image Cache works

    master

    Kube-Fledged implements a distributed Cluster Image Cache using a Kubernetes Custom Resource Definition (CRD) of kind ImageCache. Instead of a centralized registry mirror, images are pulled directly onto specific worker nodes.

    When a Pod is scheduled, if its imagePullPolicy is set to Never or IfNotPresent, the container runtime will use the image already present in the node's local cache, eliminating the latency of pulling from a remote registry. This is particularly useful for edge computing or scenarios requiring near-instant Pod startup.

    Key behaviors:

    • Node Selection: Images are distributed to nodes based on nodeSelector labels.
    • Automatic Refresh: A background controller periodically checks if images are missing (e.g., due to Kubelet's garbage collection or new nodes being added) and re-pulls them.
    • On-demand Refresh: Users can trigger a manual refresh by annotating the ImageCache resource.
    • Purging: Users can delete images from the cache via the API, provided no active containers are using them.
    # Concept: ImageCache resource manages distributed node caching
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
spec:
  cacheSpec:
    - images: ["nginx:1.15.5"]
      nodeSelector: zone=asia-south1-a

  4. Prerequisites for kube-fledged

    master

    Before installing kube-fledged, ensure you meet the following requirements:

    • A functioning Kubernetes cluster (e.g., minikube or production).
    • Cluster-admin privileges for deployment.
    • All master and worker nodes must have the kubernetes.io/hostname label.

    For development purposes, you will also need:

    • git, make, go, docker engine (>= 19.03), openssl, kubectl, helm, gpg, and gnu-sed installed on a Linux or Mac machine, with kubectl configured to access your cluster.

  5. Install kube-fledged using Helm operator

    master

    To install kube-fledged using the Helm operator into the kube-fledged namespace:

    1. Clone the repository:
      git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged
cd $HOME/kube-fledged
    2. Deploy the operator and kube-fledged:
      make deploy-using-operator
      (Note: You can deploy to a different namespace by exporting the KUBEFLEDGED_NAMESPACE variable before running the make command.)
    3. Verify the deployment:
      kubectl get pods -n kube-fledged -l app.kubernetes.io/name=kube-fledged
kubectl get imagecaches -n kube-fledged
    git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged
cd $HOME/kube-fledged
make deploy-using-operator
kubectl get pods -n kube-fledged -l app.kubernetes.io/name=kube-fledged

  6. Create an image cache

    master

    To create an image cache, define an imagecache resource. You can use the sample manifest located at deploy/kubefledged-imagecache.yaml as a template. If your images are in a private repository, ensure you add imagePullSecrets to the manifest.

    Example command:

    kubectl create -f deploy/kubefledged-imagecache.yaml

    To verify the creation:

    kubectl get imagecaches -n kube-fledged
    kubectl create -f deploy/kubefledged-imagecache.yaml
kubectl get imagecaches -n kube-fledged

  7. Build and Deploy kube-fledged from source

    master

    To build and deploy your own version of kube-fledged:

    Build

    1. Clone the repository and enter the directory.
    2. (Optional) If behind a proxy, export HTTP_PROXY and HTTPS_PROXY.
    3. Set the following environment variables for your registry and version:
      • RELEASE_VERSION: your tag
      • CONTROLLER_IMAGE_REPO: docker.io/<your_dockerhub_username>/kubefledged-controller
      • WEBHOOK_SERVER_IMAGE_REPO: docker.io/<your_dockerhub_username>/kubefledged-webhook-server
      • CRI_CLIENT_IMAGE_REPO: docker.io/<your_dockerhub_username>/kubefledged-cri-client
      • OPERATOR_IMAGE_REPO: docker.io/<your_dockerhub_username>/kubefledged-operator
    4. Run the build command:
      docker login -u <username> -p <password>
export DOCKER_CLI_EXPERIMENTAL=enabled
make install-buildx && make release-amd64

    Deploy

    1. Edit the manifests in kube-fledged/deploy:
      • In kubefledged-deployment-controller.yaml, set the image field to your controller image.
      • In kubefledged-deployment-webhook-server.yaml, set the image field to your webhook server image.
      • If using a private registry, add imagePullSecrets to kubefledged-deployment-controller.yaml.
    2. Deploy using:
      make deploy-using-yaml
    export RELEASE_VERSION=<your_tag>
export CONTROLLER_IMAGE_REPO=docker.io/<your_dockerhub_username>/kubefledged-controller
export WEBHOOK_SERVER_IMAGE_REPO=docker.io/<your_dockerhub_username>/kubefledged-webhook-server
export CRI_CLIENT_IMAGE_REPO=docker.io/<your_dockerhub_username>/kubefledged-cri-client
export OPERATOR_IMAGE_REPO=docker.io/<your_dockerhub_username>/kubefledged-operator
docker login -u <username> -p <password>
export DOCKER_CLI_EXPERIMENTAL=enabled
make install-buildx && make release-amd64

  8. Install kube-fledged using Helm chart

    master

    To install the latest version of kube-fledged using Helm:

    1. Create the target namespace:
      export KUBEFLEDGED_NAMESPACE=kube-fledged
kubectl create namespace ${KUBEFLEDGED_NAMESPACE}
    2. Add the repository and install the chart:
      helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
helm repo update
helm install kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait
    export KUBEFLEDGED_NAMESPACE=kube-fledged
kubectl create namespace ${KUBEFLEDGED_NAMESPACE}
helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
helm repo update
helm install kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

  9. Install kube-fledged using YAML manifests

    master

    To install kube-fledged into a dedicated kube-fledged namespace using pre-built images from Docker Hub:

    1. Clone the repository:
      git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged
cd $HOME/kube-fledged
    2. Deploy using the provided Makefile:
      make deploy-using-yaml
    3. Verify the deployment:
      kubectl get pods -n kube-fledged -l app=kubefledged
kubectl get imagecaches -n kube-fledged
      (Note: The second command should return 'No resources found' if the installation is successful and no caches have been created yet.)
    git clone https://github.com/senthilrch/kube-fledged.git $HOME/kube-fledged
cd $HOME/kube-fledged
make deploy-using-yaml
kubectl get pods -n kube-fledged -l app=kubefledged

  10. Remove kube-fledged from the cluster

    master

    Depending on how you installed kube-fledged, use the corresponding command to remove it:

    • If installed via YAML manifests:
      make remove-kubefledged
    • If installed via Helm chart:
      helm delete kube-fledged -n ${KUBEFLEDGED_NAMESPACE}
    • If installed via Helm Operator:
      make remove-kubefledged-and-operator
    # YAML
make remove-kubefledged

# Helm
helm delete kube-fledged -n ${KUBEFLEDGED_NAMESPACE}

# Operator
make remove-kubefledged-and-operator

  11. Manage image cache lifecycle (Add, Remove, Refresh, Delete)

    master

    kube-fledged provides several ways to manage the lifecycle of an existing imagecache (e.g., imagecache1) via kubectl:

    Add or Remove images

    Use kubectl edit to modify the image list in the manifest. The editor will open, allowing you to add or remove image entries. Save and exit to apply changes.

    kubectl edit imagecaches imagecache1 -n kube-fledged

    Refresh image cache

    To trigger an on-demand refresh of the images in the cache, use a Kubernetes annotation:

    kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=

    Delete image cache

    Important: You must purge the images from the worker nodes before deleting the imagecache resource.

    1. Purge the images using this annotation:
      kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/purge-imagecache=
    2. Monitor the status via JSON to ensure purging is complete:
      kubectl get imagecaches imagecache1 -n kube-fledged -o json
    3. Delete the resource:
      kubectl delete imagecaches imagecache1 -n kube-fledged
    # Add/Remove
kubectl edit imagecaches imagecache1 -n kube-fledged

# Refresh
kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=

# Delete (Purge first)
kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/purge-imagecache=
kubectl delete imagecaches imagecache1 -n kube-fledged

  12. Install Kube-Fledged via Helm

    master

    To install Kube-Fledged, first create a dedicated namespace, then add the official Helm repository and install the chart. It is recommended to use the --wait flag to ensure all components are ready.

    # Create the namespace
export KUBEFLEDGED_NAMESPACE=kube-fledged
kubectl create namespace ${KUBEFLEDGED_NAMESPACE}

# Install the latest version of the kube-fledged helm chart
helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
helm repo update
helm install kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait