Fleet for ESPHome

The Web UI provides several key views for managing your ESPHome fleet:

• Devices: A table of all ESPHome configs found in /config/esphome/. Includes status, tags, firmware version, IP, and ESPHome version. The row menu (⋮) allows for logs, restarts, config history, pinning, and archiving.
• Queue: Displays all compile jobs (pending, running, succeeded, failed, or blocked). Provides live build logs and worker selection details.
• Workers: Shows all local and remote workers, their platform, tags, slot counts, and disk quota usage. Includes a Routing rules... builder to match devices to workers using tags.
• Schedules: Manages recurring (cron/daily/weekly) or one-time future OTA upgrades. Schedules are stored in the device YAML.
• Header: Contains theme toggles, a "streamer mode" (blurs secrets), the global ESPHome version selector, and a shortcut to secrets.yaml.

Fleet for ESPHome provides several advanced management features not found in the standard ESPHome builder:

• Config Versioning: Every save is a git commit, providing per-file history, side-by-side diffs, and a one-click Restore button.
• Advanced Device Table: A spreadsheet-like view with sortable/filterable columns for online status, firmware version, IP, WiFi/Ethernet, and Home Assistant entity matching.
• Visible Compile Queue: Real-time job tracking, live build logs, and a searchable history drawer for all past compiles.
• Distributed Compilation: Ability to offload builds to multiple remote workers.
• Automatic Firmware Archive: Every successful compile's .bin is stored on the server for later manual flashing or debugging.
• Version Pinning: Pin specific devices to a specific ESPHome version (e.g., for beta testing) without affecting the rest of the fleet.
• Scheduled Upgrades: Define upgrade schedules (e.g., "every Sunday at 3am") directly in the device YAML.
• Tags & Routing: Use tags (e.g., kitchen, production) to create routing rules, such as ensuring ratgdo-tagged devices only compile on windows-tagged workers.
• Bulk Operations: Perform upgrades, rebuilds, or tagging across the entire fleet at once.

Fleet for ESPHome is a management layer designed for users with large, heterogeneous ESPHome device fleets (typically 15–150+ devices). It is intended to solve the limitations of the stock ESPHome dashboard by providing bulk operations, scheduling, and distributed compilation.

Key capabilities include:

• Parallelized Fleet Upgrades: Distribute the compilation queue across multiple machines (e.g., a Proxmox VM, a Windows desktop, a macOS Mini, and a local worker) to turn hours of sequential compilation into minutes.
• Hardware Offloading: Move heavy compilation tasks away from resource-constrained Home Assistant hosts (like a Raspberry Pi) to more powerful hardware.
• Bulk Operations & Scheduling: Perform fleet-wide upgrades, schedule disruptive OTA reboots for off-hours (e.g., 3 AM), and pin specific devices to certain ESPHome versions to prevent accidental updates.
• Centralized Visibility: Use the Devices tab to audit the fleet, identifying outdated devices, failed compilations, and version pinning in a single view.
• Home Assistant Integration: Keep Home Assistant as the single source of truth by maintaining configs in /config/esphome/ and using HA-native Update entities.

The Fleet for ESPHome user interface is organized into four primary views to manage your device fleet and build infrastructure:

• Devices: Manage all ESPHome configurations. You can perform one-click upgrades, bulk upgrades (outdated, changed, selected, or everything), and edit YAML inline with autocomplete and validation. Features include pinning devices to specific ESPHome versions, tagging for routing, viewing live logs, pinging devices, and viewing fully-rendered configs (with !secret substituted and packages: flattened).
• Queue: Monitor the build pipeline. View what is currently compiling, what finished, what failed, and what is queued. Access live build logs and per-row actions like Rerun, Clear, Log, Cancel, and Download firmware.
• Workers: Manage connected build workers. View platform, online status, build slots, current jobs, and disk usage vs. quota. Use the + Connect Worker button to generate docker run or docker compose snippets. Use Routing rules… to configure fleet-wide job routing based on device and worker tags.
• Schedules: View all scheduled upgrades across the fleet, including upcoming runs, last run timestamps, and success/failure status.

Additional UI features include a dark/light theme toggle and a "streamer mode" in the header that blurs tokens and secrets for screen-sharing.

The esphome-fleet-proxmox-scaler is a Python service that autoscales distributed ESPHome build workers running as Proxmox LXC containers. It functions by monitoring the Fleet server's queue depth and reconciling a pre-provisioned pool of LXCs against that queue.

The Scaling Lifecycle:

1. Polling: Every poll_interval, the scaler calls the Fleet server's /api/v1/status endpoint to get queue_size and online_workers.
2. Discovery: It finds all LXCs in the Proxmox cluster tagged with esphome-fleet-worker.
3. Calculation: It computes the desired worker count using ceil(queue_size / target_per_worker), clamped between min_total_workers and the sum of per-node maximums.
4. Reconciliation:
   • Scaling Up: It starts stopped LXCs, prioritizing nodes with the most free capacity.
   • Scaling Down: After a cooldown_seconds period, it stops the highest-VMID running LXC on the busiest node when fewer workers are needed.

Note: The scaler does not create new LXCs on demand; it only manages the power state (start/stop) of a pool of LXCs that you must pre-provision.

Devices can be organized using key/value tags stored within the # distributed-esphome: YAML comment block in the device configuration. This allows for advanced grouping and filtering in the UI.

Tag Format

Tags are defined as a map under the tags: key within the comment block:

# distributed-esphome:
#   tags:
#     location: kitchen
#     floor: "1"
#     env: prod
#     owner: stefan

UI Capabilities

• Grouping: Use the "Group by" dropdown in the Devices table to group devices by any tag key (e.g., grouping by location).
• Filtering: Click a tag chip in a device row to add it as a filter (e.g., location=kitchen). Multiple filters are combined using AND logic.
• Bulk Operations: Use multi-select on the Devices tab to "Set tag..." or "Remove tag..." across multiple devices simultaneously.
• Editing: Open the "Edit tags..." modal from the device hamburger menu to manage individual tags.

Memory Usage

Log buffering is bounded. The system expects approximately 2000 lines (avg 1 KB per line) per worker. For 16 workers, this consumes ~32 MB of memory on both the worker and the server.

Latency

There may be a delay of up to 10 seconds between opening the log dialog and seeing the first live frame (occurring if the user opens the dialog immediately after a heartbeat). The UI displays a "waiting..." hint during this period.

Error Handling

• Push Failures (5xx): If the log push endpoint returns a 5xx error, the pusher retries the next tick using the same acked_offset. Lines are not dropped on the happy path.
• Persistent Failures: If 5xx errors persist, logs are bounded by the 2000-line ring buffer; the oldest lines will fall off silently.
• Offline Workers: If a worker is offline while being "watched," the server's is_watched state remains true, but no new pushes arrive. Subscribers will see the last hydrated snapshot but no new data.
• Server Restarts: Since the WorkerLogBroker is in-memory, _next_offset and _buffers are lost on server restart. The first push after a restart is treated as a new stream (offset 0).

Fleet for ESPHome is designed for 'tech-curious' homeowners who have outgrown the standard ESPHome dashboard. The primary user (the 'Pat' persona) typically manages a mature Home Assistant OS installation with 30–100+ ESPHome devices.

Key characteristics of the target user:

• Technical Level: Comfortable with Docker, reading tracebacks, and modifying YAML. They are not necessarily professional software engineers but are technically inclined (e.g., engineers, tradespeople, or STEM professionals).
• Environment: Primarily uses Home Assistant OS (or Supervised). They rely on the Home Assistant Supervisor for add-on management and Ingress for UI access.
• Hardware Profile: Often manages a heterogeneous fleet of hardware (e.g., Raspberry Pis, mini-PCs, and idle gaming PCs) used as build workers.
• Workflow: Uses the latest stable ESPHome releases, manages secrets via secrets.yaml, and accesses their dashboard via tunnels like Nabu Casa, Tailscale, or Cloudflare.
• Use Case: Managing a single household fleet. Multi-site or shared-fleet management is not a primary use case.

The Proxmox LXC autoscaler is a Python service that manages a pool of distributed-esphome workers running as Proxmox LXC containers. Instead of cloning containers on demand (which is slow and brittle), the scaler manages a fixed pool of pre-provisioned LXCs.

Key Mechanics:

• Scaling Trigger: The scaler monitors the Fleet server's queue depth. When the queue grows, it starts LXCs; when the queue is empty, it shuts them down.
• Lifecycle: The scaler uses a start/stop loop. It performs a graceful shutdown (via Proxmox API) rather than a hard stop to allow workers to finish current ESPHome compiles.
• Discovery: The scaler identifies target LXCs using a Proxmox tag (default: esphome-fleet-worker). It discovers nodes via the Proxmox API, meaning the operator manages counts of workers per node rather than specific VMIDs.
• Resilience: If the Fleet server or Proxmox API is unreachable, the scaler logs the error and skips the reconciliation tick to prevent accidental mass-shutdowns during network blips.
• Cooldown: Scale-up is immediate. Scale-down is gated by a cooldown_seconds period (default 600s) to prevent rapid oscillation.