DOCSEARCHbyAlgolia
Submit Docs

The Watchman

repository·main·Indexed 19 days ago

https://github.com/dummylabs/thewatchman

A Home Assistant custom integration and CLI tool designed to proactively identify missing or renamed entities and services within YAML configuration files. It scans automations, scripts, dashboards, and templates using regex-based heuristics to prevent broken automations, providing reports via text files, notifications, and diagnostic sensors.

Tokens
6.3K
Snippets
8
Records
28
Agent score
71%

What's inside thewatchman

  1. Migrate User Configuration (`ConfigEntry`)

    main

    User settings follow standard Home Assistant migration logic.

    • Implementation: Managed via async_migrate_entry in __init__.py.
    • Versioning: Controlled by CONFIG_ENTRY_VERSION and CONFIG_ENTRY_MINOR_VERSION in const.py.
    • Migration Strategy:
      • Minor bumps: Use for backward-compatible additions, such as adding a new option with a default value.
      • Major bumps: Use for breaking changes that require structural transformations.

  2. How Watchman manages cache updates and re-parsing

    main

    To prevent stale data without constant disk I/O, Watchman uses an Event-Driven re-parsing model. While the configuration is cached in memory, the cache is discarded and rebuilt in the background when the following events are detected:

    • Home Assistant Service Events: Specifically call_service events like homeassistant.reload_core_config or automation.reload.
    • File System Events: Changes to configuration.yaml or valid sub-directories (where supported by the environment).

  3. How Watchman handles configuration parsing

    main

    Watchman uses a Naive Parsing Strategy instead of a formal YAML parser.

    Key Characteristics:

    • Regex-based: Files are treated as raw text streams. Regular Expressions are used to identify patterns matching domain.object_id.
    • Performance: Regex processing is faster than building a DOM for large YAML files.
    • Robustness: It avoids crashes caused by custom Home Assistant tags (e.g., !include, !secret) or Jinja2 templates that standard YAML parsers cannot process.
    • Fault Tolerance: Syntax errors in YAML files do not crash the auditor; the regex approach simply skips problematic lines and continues scanning.

  4. How the parser handles Jinja2 templates and dynamic strings

    main

    The parser is designed to extract entities even when they are embedded within dynamic Jinja2 templates or complex YAML structures.

    Template Detection

    • Inline Markers: The parser detects template markers such as {{, {%, {#, and [[[ anywhere within a string. This allows it to identify entities in strings like action: domain.service_{{ id }}.
    • Suppression Bypassing: While certain keys like trigger:, triggers:, and condition: normally suppress their immediate string values to avoid false positives, this suppression is bypassed if a Jinja2 template marker is detected (e.g., condition: "{{ is_state('light.x', 'on') }}" will correctly yield light.x).

    Complex YAML Structures

    • Block Scalars: For multi-line strings (using | or >), the parser applies line offset correction so that reported line numbers match the actual content. For action or service keys using block scalars, it performs line-by-line analysis: lines with only a service ID are treated as services, while lines with template syntax are scanned for entities.
    • Embedded Services: Service calls embedded within string values (e.g., inside a template) are detected using regex patterns (e.g., service: domain.service).

  5. How Watchman works

    main

    Watchman is a Home Assistant integration that scans your YAML configuration files (automations, scripts, dashboards, templates, etc.) to find references to entities (sensors, timers, etc.) and services/actions. It then verifies if these entities are currently available and if the services exist in the Home Assistant registry.

    Key Mental Models:

    • Regex-based Heuristics: Watchman uses lightweight regex to detect references rather than building a full configuration model. This means it may produce false positives (detecting things that aren't real entities) or false negatives (missing real references).
    • Scope: It only reports entities and services that it can find referenced in your configuration. It does not perform a general scan of all missing entities in your instance.
    • Silencing Noise: You can use the Ignored entities and actions or Ignored labels options to suppress false positives.

  6. Migrate Operational Statistics (JSON Store)

    main

    Operational statistics are stored using homeassistant.helpers.storage.Store.

    • Implementation: Managed via WatchmanCoordinator in coordinator.py.
    • Versioning: Controlled by STORAGE_VERSION.
    • Migration Strategy:
      • Soft Upgrades: The system relies on Python's dict.get(key, default) method for backward compatibility.
      • Reset: If STORAGE_VERSION is incremented, Home Assistant treats the old file as invalid and starts with fresh statistics (an effective reset).

  7. How Watchman tracks entity ownership via Context Propagation

    main

    Watchman uses Recursive Context Propagation to determine the "ownership" of an entity (e.g., whether light.kitchen belongs to an automation, a script, or a group) without a full DOM parser.

    The ParserContext Object

    An immutable ParserContext data class is passed down during recursive traversal, containing:

    • parent_type: The container type (e.g., automation, script, group).
    • parent_id: The unique identifier (e.g., automation.turn_on_lights).
    • parent_alias: The human-readable name.
    • is_active: A boolean flag indicating if the parser is currently inside a defined context.

    Context Locking Strategy

    To prevent nested structures (like repeat, choose, or if blocks) from being misinterpreted as new top-level entities, Watchman applies a Top-Down Locking Strategy:

    • If is_active is True, the parser strictly forbids creating a new context.
    • Entities found within these nested blocks correctly inherit the ownership of the parent context.

  8. How Purpose-specific Condition Intents are handled

    main

    To support Home Assistant 2025.12+ features, the parser treats string values appearing directly under a condition: key (e.g., condition: person.is_not_home) as Purpose-specific condition intents rather than entity references. These are suppressed to prevent false positives.

    Exception: This suppression is bypassed if the value contains a Jinja2 template marker. For example:

    • condition: person.is_not_home $\rightarrow$ Ignored (treated as intent).
    • condition: "{{ is_state('light.x', 'on') }}" $\rightarrow$ Extracted (light.x is identified because of the template marker).

  9. Migrate the SQLite Parsing Index

    main

    The SQLite database (.storage/watchman_v2.db) stores the parsing cache. Because this data is reproducible via rescanning, the system prioritizes stability over data preservation.

    • Implementation: Managed via WatchmanParser._init_db in parser_core.py.
    • Versioning: Uses SQLite PRAGMA user_version, controlled by CURRENT_DB_SCHEMA_VERSION.
    • Migration Strategy (Hard Reset): If the database schema version does not exactly match the code's expected version (vCurrent != vTarget), the .db file is deleted and rebuilt from scratch. The cache is treated as disposable.

  10. Understand the parser heuristics for entity extraction

    main

    The Watchman uses a set of 24 heuristics to identify Home Assistant entities and services while minimizing false positives. These heuristics govern how the parser handles different file types, YAML structures, and template syntaxes.

    Key Extraction Logic

    • Domain Validation: Extracted strings must start with a valid Home Assistant domain.
    • Prefix Handling: The parser automatically strips the states. prefix (e.g., states.light.living_room becomes light.living_room).
    • Context Awareness: The parser distinguishes between automation (containing trigger+action) and script (containing sequence) contexts. It also uses Context Locking to ensure entities inside control flows (like repeat, choose, or if) are attributed to the top-level automation/script rather than the control flow name.
    • File Type Detection: The parser applies specific logic based on whether it detects Home Assistant YAML, JSON, or ESPHome YAML. Note that .json files are explicitly ignored to prevent false positives.
    • ESPHOME Specifics: In ESPHome files, entities and services are only extracted if they are values of service, action, or entity_id keys.

    False Positive Prevention (Ignored Patterns)

    To avoid misidentifying non-entities, the parser ignores:

    • Suffixes & Wildcards: Underscore suffixes (_) and immediate wildcards (*).
    • Function Calls: Identifiers followed by an opening parenthesis (.
    • String Concatenation: Entities part of dynamic templates using +, ~, %, or .format.
    • Metadata Keys: Content under url, example, and description keys.
    • Boundary Violations: Matches preceded by path separators (/, \), hyphens (-), pipes (|), or special symbols (@, $, %, &), or followed by hyphens (-), dots (.), curly braces ({), or square brackets ([).
    • Bundled Ignores: Known system strings like timer.cancelled, date.*, and event.*.

  11. Understand the Watchman data storage model

    main

    Watchman uses a hybrid storage approach across three distinct mechanisms to balance performance and reliability. Understanding where data lives is critical for managing persistence and migrations.

    Data TypeStorage MechanismLocationDescription
    User ConfigurationHA Config Entries.storage/core.config_entriesPersistent settings (ignore lists, report paths). Managed by HA.
    Parsing IndexSQLite.storage/watchman_v2.db"Cold" cache. Relational data (files, entities, line numbers). Optimized for read performance.
    Operational StatsJSON Store.storage/watchman.stats"Hot" data. Volatile statistics (last scan time, duration, counters). Loaded into memory on startup.

  12. How Watchman's Scan → Analyze → Report cycle works

    main

    Watchman operates using a three-stage lifecycle to identify "orphan" entities (entity IDs referenced in YAML/dashboards that no longer exist in the Home Assistant state machine):

    1. Scan: Upon initialization, Watchman performs a "deep scan" by traversing the configuration directory and parsing files line-by-line using Regular Expressions to build a global set of referenced entities.
    2. Analyze: The system stores these entities in an in-memory cache. When a check is triggered, Watchman compares this cached list against the current Home Assistant State Machine.
    3. Report: Any entity found in the cache but missing from the State Machine is flagged and sent to the Reporter for output (e.g., text files or notifications).

    This model ensures that the heavy I/O of file reading is decoupled from the logic of checking states, making the actual audit command nearly instantaneous.