Cloud Foundry BOSH

Bosh Monitor supports several plugins for event handling and system management:

• DataDog: Sends events to DataDog.com via API.
• Consul Event Forwarder: Forwards NATS heartbeats and alerts to Consul.
• Event Logger: Logs all events.
• PagerDuty: Sends events to PagerDuty.com via API.
• Resurrector: Automatically restarts VMs that have stopped sending heartbeats.
• EventLogger: Stores events directly in the Director DB.

Cloud Foundry BOSH is an open source tool designed for release engineering, deployment, lifecycle management, and monitoring of distributed systems.

This specific repository is a Bosh Release. It contains the binaries and configuration templates required to deploy a new Bosh Director instance. This deployment is typically driven by a Bosh deployment manifest and executed via a Bosh CLI invocation or an existing Bosh Director instance.

The s3cli package provides a command-line interface for interacting with S3-compatible blobstores used by BOSH. It is a compiled version of the bosh-s3cli project.

To use this tool, you can download the pre-compiled binaries from the official S3 artifact bucket.

BOSH Monitor is a component that listens to and responds to events (Heartbeats & Alerts) on the NATS message bus. It serves as a central hub for monitoring the health of BOSH agents and managing alerts.

Its core functionality is driven by several primary components:

• Agent Monitor: Maintains a record of known agents by subscribing to heartbeat events.
• Director Monitor: Maintains a record of known agents by periodically polling the Director via HTTP.
• Agent Analyzer: Periodically analyzes agent state to generate "Agent Missing" alerts if heartbeats are not received within configured time periods.
• Event Handling Plugins: Generic plugins that process incoming Heartbeats and Alerts.

IP allocations are managed by an IP provider, specifically using the DatabaseIpRepo.

Key behaviors of DatabaseIpRepo:

• Shared State: IP reservations are stored in a database and are shared across multiple deployments when using cloud config.
• Uniqueness: It ensures global uniqueness. If you attempt to re-reserve an IP that is already in use, DatabaseIpRepo verifies if the IP belongs to the same instance. If it belongs to a different instance, it throws a NetworkReservationAlreadyInUse error.

When developing changes related to specific infrastructure or a CPI, you may need to build a local stemcell.

Important for local development:

• DO NOT set CANDIDATE_BUILD_NUMBER. This allows you to build a stemcell of version 0000, which BOSH rake tasks recognize as a local stemcell.
• Place the generated stemcells of version 0000 into the bosh/tmp directory before running Acceptance Tests (BATs).

In BOSH, the instance is the owner of a given IP address. This ownership persists even if the underlying VM is lost or detached.

• Persistence: As long as an instance exists, its IP address remains reserved and unavailable to other instances (even from different deployments). If an instance's VM is recreated, it will receive the same IP address.
• Release: Deleting an instance (e.g., via scaling down or removing an availability zone) results in all its IPs being released.
• Compilation Instances: IPs used by compilation instances are released automatically once compilation is finished and the instance is deleted.

BOSH Monitor uses two mechanisms to track agents:

1. Agent Monitor (Heartbeat-based): Listens for hm.agent.heartbeat.<agent_id> on NATS. If an agent is known, its last heartbeat timestamp is updated. If unknown, it is recorded as a "rogue agent".
2. Director Monitor (Polling-based): Periodically polls the Director via HTTP to discover managed VMs.
   • Endpoint: GET /deployments/<deployment_name>/vms
   • Response: JSON containing agent IDs, job names, and indices.

Discovery Logic:

• If a new agent is discovered via polling, it is recorded as part of the managed deployment.
• If a "rogue agent" is discovered via polling, its "rogue agent" flag is cleared.
• The Director Monitor does not poll agents directly; it reconciles the state reported by the Director with the state reported by heartbeat events.