DOCSEARCHbyAlgolia
Submit Docs

Homelab Autoscaler

website·Indexed 31 days ago

https://autoscaler.homecluster.dev

A Kubernetes operator that adds dynamic scaling to physical homelab clusters by managing physical machine power states based on workload demands. It utilizes a finite state machine (FSM) for reliable node power management and supports wake-on-LAN, IPMI, and BMC. The system includes Custom Resource Definitions (CRDs) for Groups and Nodes to define autoscaling policies.

Tokens
17.7K
Snippets
73
Records
108
Agent score
52%

What's inside Homelab Autoscaler

  1. Overview of Homelab Autoscaler

    Homelab Autoscaler provides dynamic scaling for physical homelab Kubernetes clusters. Unlike cloud providers that create virtual machines, it manages the power state of physical machines based on workload demands to minimize power consumption and handle resource bursting.

  2. Inspect homelab-autoscaler resource details and events

    Use describe to get detailed information about groups and nodes, and check the event log sorted by timestamp to identify recent failures.
    # Describe resources for detailed information
kubectl describe group <group-name> -n homelab-autoscaler-system
kubectl describe node.infra.homecluster.dev <node-name> -n homelab-autoscaler-system

# Check events
kubectl get events -n homelab-autoscaler-system --sort-by='.lastTimestamp'

  3. Understand the E2E test architecture components

    The E2E test environment consists of the following components:

    1. VM Control Server (examples/k3d/vm_control_server.py): An HTTP server (port 8080) that simulates VM power actions via /start?vm=VM_NAME and /stop?vm=VM_NAME by executing k3d node start/stop commands.
    2. Homelab Autoscaler Controller (cmd/main.go): Runs locally, exposing a gRPC server on port 50052 to implement the Cluster Autoscaler integrations API.
    3. Kubernetes Cluster Autoscaler: Deployed via Helm in the homelab-autoscaler-system namespace; it makes scaling decisions based on pod demand.
    4. Node Custom Resources (examples/k3d/nodes1.yaml): Define nodes with startupPodSpec (calls /start) and shutdownPodSpec (calls /stop) to trigger VM state changes.
    5. Service Forwarders:
      • homelab-autoscaler-grpc-local: Routes cluster traffic (50052) to the local gRPC server.
      • host-internal: Routes traffic (8080) to the host's VM control server.
    6. Test Utilities (test/utils/utils.go): Provides helpers for node tainting, gRPC log verification, and monitoring node/pod states.

  4. Homelab Autoscaler Architecture Overview

    Homelab Autoscaler is a Kubernetes operator designed for physical node autoscaling. It acts as a bridge between the Kubernetes Cluster Autoscaler and physical hardware. It implements the Cluster Autoscaler CloudProvider interface via a gRPC server, allowing the Cluster Autoscaler to trigger power operations on physical nodes through Kubernetes Jobs.

  5. Understand Homelab Autoscaler node power state transitions

    Homelab Autoscaler uses a finite state machine (FSM) based on the looplab/fsm library to manage node power states. The system moves through the following states:

    • Shutdown: Node is powered off, ready for startup.
    • StartingUp: Node startup operation is in progress.
    • Ready: Node is powered on and operational.
    • ShuttingDown: Node shutdown operation is in progress.

    Startup Flow:

    1. StartNode event triggers transition: Shutdown $\rightarrow$ StartingUp.
    2. A Kubernetes Job is created to perform the startup.
    3. JobCompleted event triggers transition: StartingUp $\rightarrow$ Ready.
    4. JobFailed event triggers transition: StartingUp $\rightarrow$ Shutdown (for retry).

    Shutdown Flow:

    1. ShutdownNode event triggers transition: Ready $\rightarrow$ ShuttingDown.
    2. A Kubernetes Job is created to perform the shutdown.
    3. JobCompleted event triggers transition: ShuttingDown $\rightarrow$ Shutdown.
    4. JobFailed event triggers transition: ShuttingDown $\rightarrow$ Ready (for retry).

  6. Understand Homelab Autoscaler communication flow

    The system relies on two primary communication paths:

    1. gRPC Flow: The in-cluster Cluster Autoscaler communicates with the Local Controller (outside the cluster) via a Service Forwarder on port 50052.
    2. Node Lifecycle Flow: Startup/Shutdown pods send HTTP requests (/start?vm=X or /stop?vm=X) to the VM Control Server on port 8080, which then executes k3d node start/stop commands via the k3d CLI.

  7. Understand the Homelab Autoscaler architecture

    The system consists of several core components:

    • Controllers: The Group Controller manages policies, the Node Controller handles power operations, and the Core Controller bridges Kubernetes nodes with Node CRDs.
    • gRPC Server: Implements the standard Cluster Autoscaler CloudProvider interface to allow the Kubernetes Cluster Autoscaler to request node scaling.
    • Scaling Process:
      • Scale Up: Triggered by resource demand $\rightarrow$ Cluster Autoscaler requests nodes via gRPC $\rightarrow$ System powers on physical machines $\rightarrow$ Machines join cluster.
      • Scale Down: Triggered by low utilization $\rightarrow$ Cluster Autoscaler identifies unneeded nodes $\rightarrow$ Pods are drained $\rightarrow$ System powers off physical machines.

  8. Define autoscaling policies using the Group CRD

    The Group Custom Resource Definition (CRD) is used to define autoscaling policies for collections of physical nodes. It specifies thresholds for scaling down based on CPU and GPU utilization, timing parameters for node removal, and provisioning timeouts.
    apiVersion: infra.homecluster.dev/v1alpha1
kind: Group
metadata:
  name: worker-nodes
  namespace: homelab-autoscaler-system
spec:
  name: worker-nodes
  maxSize: 5
  nodeSelector:
    node-type: worker
    zone: homelab
  scaleDownUtilizationThreshold: 0.5
  scaleDownGpuUtilizationThreshold: 0.5
  scaleDownUnneededTime: "10m"
  scaleDownUnreadyTime: "20m"
  maxNodeProvisionTime: "15m"
  zeroOrMaxNodeScaling: false
  ignoreDaemonSetsUtilization: true

  9. Manage physical machine power states with HomeCluster

    HomeCluster allows for the management of physical machine power states based on current workload demands. It provides support for several hardware interfaces including wake-on-LAN, IPMI, BMC interfaces, and smart PDUs.

  10. Homelab-autoscaler core infrastructure capabilities

    The core infrastructure of the homelab-autoscaler is operational and includes the following components:

    • Custom Resource Definitions (CRDs): Full implementation of Group and Node CRDs.
    • Group Controller: Manages autoscaling policies and group health.
    • Core Controller: Ensures consistency between Kubernetes nodes and Node CRDs.
    • Node Controller: Handles state management via a Finite State Machine (FSM) and power operations.
    • Webhook System: Provides validation and mutation webhooks for CRDs.
    • Helm Chart: Supports deployment with automated CRD synchronization.

  11. E2E test project structure and file reference

    The E2E testing suite for the Homelab Autoscaler consists of the following components:

    Orchestration & Setup:

    • Makefile: Defines test-e2e and run-e2e targets.
    • development/install-dev.sh: Sets up the environment, creates the cluster, and installs Helm.
    • examples/k3d/create-cluster.sh and examples/k3d/delete-cluster.sh: Scripts for cluster lifecycle management.

    Implementation & Fixtures:

    • test/e2e/k3d_integration_test.go: Contains the main sequential test logic.
    • test/utils/utils.go: Provides helper functions for VM control, node management, and job verification.
    • examples/k3d/nodes1.yaml: Defines Node Custom Resources (e.g., agent-0, agent-1).
    • examples/k3d/hello-world-deployment.yaml: Defines the test workload (nginx deployment).
    • examples/k3d/vm_control_server.py: An HTTP server used to simulate node power control.

    Runtime Artifacts:

    • ./kubeconfig: The cluster configuration file generated by install-dev.sh.
    • /tmp/vm-control-requests.log: Logs for the VM control server.
    • /tmp/local-server.pid and /tmp/vm-control.pid: PID files for the local and VM control servers.

  12. Run the homelab-autoscaler controller locally

    The controller can be run locally using the Makefile or directly via go run to provide custom configuration flags for metrics, leader election, and the gRPC server.
    # Run with development settings
make run

# Run with custom flags
go run cmd/main.go \
  --metrics-bind-address=:8080 \
  --leader-elect=false \
  --enable-grpc-server=true \
  --grpc-server-address=:50051