DOCSEARCHbyAlgolia
Submit Docs

Bottlerocket Update Operator (Brupop)

repository·develop·Indexed 42 days ago

https://github.com/bottlerocket-os/bottlerocket-update-operator

A Kubernetes operator that automates and coordinates Bottlerocket OS updates across a cluster using a controller-agent architecture. It manages the update lifecycle through a controller for orchestration, a DaemonSet agent for node-level updates, and an API server for secure cluster object mutations. Brupop utilizes 'Bottlerocket shadows' as reflections of nodes to interact with the host's apiclient update interface.

Tokens
30.5K
Snippets
85
Records
130
Agent score
52%

What's inside Brupop

  1. What is the Bottlerocket Update Operator (Brupop)?

    develop

    The Bottlerocket update operator (Brupop) is a Kubernetes operator designed to coordinate Bottlerocket updates across hosts in a cluster. It manages the update lifecycle using three main components:

    1. Controller: A deployment running on a single node that orchestrates updates across the entire cluster.
    2. Agent: A DaemonSet running on every Bottlerocket node. The agent periodically queries for updates, drains the node, and performs the update when instructed by the controller.
    3. Update Operator API Server: A deployment that handles cluster object mutations. To ensure security, it uses the Kubernetes TokenReview API to verify that requests are actually coming from the agent pod running on the specific node making the request.

    Key Concepts:

    • Security: The API Server requires cert-manager to use CA certificates for SSL communication with agents.
    • Update Waves: Updates are rolled out in waves to minimize cluster-wide impact, meaning nodes may not update simultaneously.

  2. How Brupop coordinates Bottlerocket updates

    develop

    Brupop (Bottlerocket Update Operator) automates Bottlerocket updates across a Kubernetes cluster while minimizing workload disruption. It uses a two-component architecture:

    1. Controller: A central component that manages the desired state of the fleet. It uses a BottlerocketShadow Custom Resource (CR) associated with each node to determine what actions should be taken.
    2. Agent: A DaemonSet running on every Bottlerocket node. The agent acts as a local control loop that reads the desired state from the BottlerocketShadow CR and drives the local node toward that state (e.g., performing the actual update and reboot).

    Key Workflow:

    • The controller identifies nodes to manage and sets a desired state in their respective BottlerocketShadow resources.
    • The agent observes the BottlerocketShadow resource, executes the necessary local Bottlerocket API calls, and updates the resource's status to reflect the node's current state.
    • To prepare nodes for updates, Brupop utilizes Kubernetes Cordon and Drain APIs. This ensures that pods are gracefully moved off the node before the update and reboot occur.

  3. Monitor Brupop updates with Prometheus

    develop

    Brupop integrates with Prometheus to provide observability into the update process. The operator exposes an HTTP endpoint that allows Prometheus to scrape metrics regarding the operator's actions.

    Available Metrics:

    • State Transitions: Metrics tracking how nodes move through the update state machine.
    • Version Tracking: Information regarding the version of a node before and after an update is completed.

    This allows you to build dashboards to visualize the update status of your fleet and track the progress of rolling updates across your cluster.

  7. How Brupop authorizes Bottlerocket node updates

    develop

    To prevent a single node from having permissions to modify the BottlerocketShadow custom resource of any other node, Brupop uses a specialized authorization flow:

    1. Token Projection: Each DaemonSet pod uses Service Account Token Volume Projection to receive a unique token.
    2. API Request: The node sends update requests for its BottlerocketShadow resource to the Brupop controller via a web API.
    3. Validation: The controller uses the Kubernetes TokenReview API to validate the token.
    4. Identity Check: The controller verifies that the calling pod resides on the specific node associated with the target custom resource before allowing the update.

  8. Configure Brupop behavior via Custom Resources

    develop

    Brupop uses Custom Resources (CRs) for both node-level management and global configuration:

    • BottlerocketShadow: A CR associated with each managed Bottlerocket node. It contains the node's status information and the spec (desired state) set by the controller. The agent uses this to drive local system changes.
    • Operator Configuration CR: A CR associated with the Brupop operator itself. This allows you to configure cluster-wide settings, such as enabling/disabling Brupop or enabling optional features like integration with AWS AutoScaling groups (to prevent node termination during reboots).

  9. Understand the lifecycle of Brupop Custom Resources

    develop

    Brupop creates per-node Custom Resources to describe the state of individual nodes. These resources use Kubernetes ownerReferences to link them to the specific Node they describe.

    Because of this association, Kubernetes garbage collection will automatically terminate and clean up these Custom Resources when the associated Node is removed from the cluster, preventing resource leakage.

    metadata:
  name: ip-192-168-48-253.us-west-2.compute.internal
  ownerReferences:
    - apiVersion: v1
      kind: Node
      uid: c5a4b820-39a6-46e8-9a2f-a2623a17dfbf
      name: ip-192-168-48-253.us-west-2.compute.internal

  10. Understand the BottlerocketShadow (brs) Custom Resource

    develop

    The BottlerocketShadow (short name: brs) is a Custom Resource used by Brupop to track the update status of each Bottlerocket node. It acts as the source of truth for the update state machine.

    • Spec: Contains the desired state for the node, specifically the desired Bottlerocket version and the desired state in the Brupop state machine.
    • Status: Contains the current state machine state and the desired latest version for the node.

  11. How the Brupop components (Controller, Agent, and API Server) work together

    develop

    Brupop uses a distributed architecture to manage updates safely:

    1. Controller: The cluster-wide control loop. It manages the state machine by reading the status of a BottlerocketShadow and setting the next desired state in its spec.
    2. Agent: A node-local control loop. It gathers system info via Bottlerocket's local API (communicating through a mounted unix socket) and updates the BottlerocketShadow status. The agent executes state transitions (like cordoning and draining pods) based on the spec.
    3. API Server: A privileged component that handles all writes to the Kubernetes API on behalf of the Agent. This prevents Agents from having broad RBAC permissions. The API Server validates requests using a special token and TokenReview APIs to ensure an Agent can only modify its own BottlerocketShadow object.

  12. How Brupop architecture works

    develop

    Brupop (Bottlerocket Update Operator) is a Kubernetes Operator that orchestrates Bottlerocket OS updates across a fleet of nodes using nested control loops and a custom resource called BottlerocketShadow (brs).

    Core Components

    1. controller Component: The fleet orchestrator. It watches all BottlerocketShadow resources and uses their status to determine the current state of the fleet. It manages the update lifecycle by updating the spec of brs objects to drive them toward the desired state. It currently ensures only one update happens at a time across the fleet.

    2. agent Component: The node-level executor. It runs on each Bottlerocket node and performs the following:

      • Gathers OS information and updates the brs.status.
      • Monitors brs.spec to decide when to trigger an update.
      • Performs node maintenance: cordons the Node and drains Kubernetes Pods prior to rebooting.
      • Creates a shadow brs object if one does not exist for the node.

    3. apiserver Component: A privileged web API that acts as a gatekeeper. Because Kubernetes RBAC would normally allow any agent to modify any brs object (creating a security risk), all agent writes are channeled through this apiserver. It validates that requests are coming from the specific Kubernetes Node associated with the brs object using the Kubernetes token API.

    The BottlerocketShadow (brs) Custom Resource

    Brupop uses the BottlerocketShadow (or brs) Custom Resource to track the state of each Bottlerocket node. It follows the standard Kubernetes pattern:

    • spec: Defines the desired state (e.g., the target version or update stage).
    • status: Reflects the current state of the node (e.g., current OS version, update progress) as reported by the agent.