DOCSEARCHbyAlgolia
Submit Docs

Home Assistant Statistics Custom Integration

repository·main·Indexed 41 days ago

https://github.com/klausj1/homeassistant-statistics

A database-agnostic custom integration for Home Assistant that allows importing and exporting long-term statistics using CSV, TSV, or JSON formats. It provides actions such as import_from_file, import_from_json, export_statistics, and export_inventory to manage statistic data via the Home Assistant UI, automations, or REST API.

Tokens
16.1K
Snippets
35
Records
81
Agent score
52%

What's inside Home Assistant Statistics

  1. Understand counter statistics: state, sum, and delta

    main

    When working with counter-type entities (like energy meters), three key values are used to track data:

    TermDescriptionExample
    stateThe actual meter reading at the end of the period.6913.045 kWh
    sumThe cumulative value since the counter was created.6913.045 kWh (if started at 0)
    deltaThe change in the sum during the previous hour.0.751 kWh

    Key Relationship: The difference between state and sum is the initial value of the counter when it was first connected to Home Assistant. This difference remains constant as long as the counter is functioning correctly.

  2. How statistics are categorized by the exporter

    main

    The exporter classifies statistics (not entities) by checking the statistic_id against Home Assistant's internal data sources.

    Classification Logic

    CategoryConditionInterpretation
    Externalsource != "recorder" OR statistic_id contains :Not an internal entity-based statistic.
    Activesource == "recorder" AND no : in statistic_id AND exists in entity_registry.entitiesThe entity is currently registered in HA.
    Orphansource == "recorder" AND no : in statistic_id AND exists in entity_registry.deleted_entitiesThe entity was removed, but a deleted record remains.
    Deletedsource == "recorder" AND no : in statistic_id AND exists in neither registry collectionThe statistic exists in the database, but HA has no registry knowledge of the entity.

  3. How statistics import and export work

    main

    The integration operates using two distinct modes for statistics:

    • Internal Statistics (Recorder Source): These use the standard Home Assistant entity format (e.g., sensor.living_room_temperature). The integration validates that the entity exists and matches the units provided in the import file.
    • External Statistics (Custom Source): These use a domain:name format. This allows you to create synthetic statistics that do not correspond to an existing Home Assistant entity.

    Import Logic

    • Normal Import: Direct extraction of mean/min/max, sum/state, or delta values from the file.
    • Delta Import: If your file contains a delta column, the integration performs a two-pass database lookup to find a reference value (either before the oldest timestamp or after the newest) to convert those deltas into absolute sum or state values.

    Export Logic

    • The export_statistics service fetches data from the Home Assistant recorder API.
    • It automatically handles timezone conversion from UTC (storage) to your local timezone.
    • It supports .json, .csv, and .tsv extensions. For counters, it automatically calculates delta columns in the exported file.

  4. Understand the Dual-Mode Statistics System

    main

    The integration distinguishes between two mutually exclusive types of statistics. Choosing the correct format determines how Home Assistant validates and stores your data.

    AspectInternal (recorder)External (custom)
    Formatsensor.name (dot separator)domain:name (colon separator)
    SourceExisting HA entityNew synthetic entity
    Unit SourceFlexible (entity or CSV)Must be from CSV (TABLE)
    ValidationEntity must exist in HADomain cannot be "recorder"
    HA APIasync_import_statistics()async_add_external_statistics()

    Use the Internal mode when you want to add statistics to an entity that already exists in your Home Assistant instance. Use the External mode when you are importing data for a new entity that does not yet exist (e.g., a synthetic entity defined by a domain and name).

  5. Understand the difference between measurement and counter statistics

    main

    Home Assistant distinguishes between two types of statistics based on their state_class:

    1. Measurement type: Uses state_class of measurement or measurement_angle. These represent instantaneous values (e.g., temperature, voltage).
    2. Counter type: Uses state_class of total or total_increasing. These represent cumulative values (e.g., energy meters, water usage).

    Note for Imports: Since v5.1.0, you can use a single import file for both types (Mixed Import). However, if you are using a mixed file, you cannot use the delta column.

  6. Understand the `import_statistics.export_inventory` service output categories

    main

    The import_statistics.export_inventory service exports an inventory of long-term statistics. Each row includes a category column that classifies the statistic based on its relationship with the Home Assistant Entity Registry and the Recorder.

    Use these categories to determine which statistics are safe to clean up:

    • Active: The statistic is internal to the recorder and the corresponding statistic_id exists in the active entity_registry.entities. Do not delete these based on the inventory alone.
    • Orphan: The statistic is internal, but the entity exists only in entity_registry.deleted_entities. These are good candidates for cleanup if you do not intend to restore the entity.
    • Deleted: The statistic is internal, but the entity is completely absent from both the active and deleted Entity Registries. These are strong candidates for cleanup.
    • External: The statistic is not an internal recorder statistic (e.g., source != "recorder" or the statistic_id contains a :). Do not treat these as deletable based on registry checks; management depends on the external integration.

  7. Understand how Home Assistant records and timestamps statistics

    main

    Home Assistant follows a specific lifecycle for recording statistics:

    1. State Changes: Recorded in the states table (sampled every 5 seconds).
    2. Short-term Statistics: Compiled every 5 minutes into statistics_short_term.
    3. Long-term Statistics: Aggregated every 60 minutes into the statistics table.

    Timestamp Logic

    Statistics use a period_start timestamp. This represents the start of the measurement hour, not the time the measurement was actually taken.

    • Example: If a value is measured at 15:00:00, it is recorded in the database with a period_start of 14:00:00.
    • Delta Calculation: The delta for a period (e.g., 14:00) is calculated as sum(14:00) - sum(13:00). This represents the consumption that occurred between 14:00 and 15:00.

  8. Understand statistic category classifications

    main

    The inventory export classifies statistics into four categories based on the Home Assistant entity registry:

    • Active: The statistic_id exists in the entity registry's active entities (core.entity_registry.entities).
    • Orphan: The statistic_id exists in the deleted entities (core.entity_registry.deleted_entities) and has a non-null orphaned_timestamp.
    • Deleted: The statistic_id is in the deleted entities list but has no orphaned_timestamp, OR it is not found in the entity registry at all.
    • External: The statistic is external (either source != "recorder" or the ID uses the domain:name format with :).

  9. How mixed-type imports work

    main

    If your import file contains both measurement statistics (e.g., min, max, mean) and counter statistics (e.g., sum, state), the system uses a "Split-then-Process" approach.

    1. Detection: The system identifies the file as ImportDataType.MIXED if both sensor and counter columns are present.
    2. Classification: Rows are grouped by statistic_id. Each entity is classified as either a SENSOR or a COUNTER based on which value columns contain non-NaN data.
    3. Processing: The data is split into two separate DataFrames. The sensor DataFrame is processed for min/max/mean, and the counter DataFrame is processed for sum/state.

    Critical Constraint: You cannot use delta columns in a mixed-type file. A file containing a delta column must only contain delta data and cannot coexist with min, max, mean, sum, or state columns.

  10. How delta processing works

    main

    The delta import follows a three-stage asynchronous pipeline to convert relative changes into absolute values:

    1. Preparation: The system detects the delta column and identifies the oldest and newest timestamps for each statistic_id.
    2. Reference Lookup: The system queries the Home Assistant database to find existing statistics at least 1 hour before the oldest timestamp (or 1 hour after if the older reference is missing).
    3. Conversion: A synchronous calculation converts the deltas into absolute values via accumulation or subtraction based on the retrieved references.

    Constraint: Like mixed imports, delta columns are incompatible with other value columns like min, max, mean, sum, or state in the same file.

  11. Import and Export Timezone Behavior

    main

    The integration handles timezones bidirectionally to ensure data integrity between local files and the UTC-based Home Assistant database.

    • Importing: The system takes timestamps in the user's local timezone (parsed via zoneinfo.ZoneInfo(timezone_id)) and converts them to UTC (.astimezone(dt.UTC)) for internal storage.
    • Exporting: The system retrieves UTC timestamps from the recorder and converts them to the user's specified timezone for display, using the user's preferred datetime format.

  12. How the Test Suites work

    main

    The project uses pytest with three distinct suites that run sequentially by default:

    1. Unit Tests (tests/unit_tests/): No dependencies. Tests pure functions like helpers and delta conversions.
    2. Integration Tests (Mock) (tests/integration_tests_mock/): Uses a mocked Home Assistant. Tests service handlers without a real HA instance.
    3. Integration Tests (Real) (tests/integration_tests/): Requires a running Home Assistant instance and a valid HA_TOKEN_DEV. Tests against a real HA environment.

    Important for First-Time Integration Tests: Integration tests create a fresh database. If you are running them for the first time, you must:

    1. Start HA manually with scripts/develop.
    2. Verify it is running at http://localhost:8123.
    3. Stop HA using Ctrl+C.
    4. Run the tests: source .env && pytest tests/integration_tests.