UK Bin Collection Data (UKBCD)

repository·master·Indexed 43 days ago

https://github.com/robbrad/ukbincollectiondata

A Python library and Home Assistant integration (version 0.172.3) that provides a standardized way to access UK council bin collection schedules in JSON format. It supports various scraping methods, including Beautiful Soup 4 and Selenium, to extract data from council websites for use in home automation.

Tokens
18K
Snippets
45
Records
72
Agent score
56%

What's inside UK Bin Collection Data

  1. Overview of UK Bin Collection Data (UKBCD)

    master
    UK Bin Collection Data (UKBCD) is a project that provides UK council bin collection data in a standardized JSON format. It is designed for use in home automation (like Home Assistant) to trigger actions, such as lighting up an LED bar on bin collection days. The project uses various scraping methods, including Beautiful Soup 4, to extract data from council websites, CSV files, or by emulating web browser behavior.
  2. Understand the manual_refresh_only configuration behavior

    master

    The configuration key manual_refresh_only can be confusing due to its naming.

    • UI Label: "Automatically refresh the sensor"
    • Behavior: When this flag is set, it controls whether the sensor updates automatically.

    Despite the name manual_refresh_only, the logic is designed to match the Home Assistant UI label. Do not attempt to invert the logic to match the key name, as this will break the expected 'automatic refresh' behavior for existing users.

  3. Understand the impact of shared HTTP fetch retries

    master

    The project implements a shared HTTP fetch retry mechanism (via PR #2155) for councils using the default requests-based fetch path (approximately 101 of the 352 councils).

    Side Effects

    • Resilience: Increases stability against transient network errors.
    • Increased Failure Latency: If a council's website is genuinely down, the integration will take longer to report a failure (up to ~46s of retry backoff plus connection time) instead of failing immediately.
    • Error Reporting: Because retries can outlast Home Assistant's per-update timeout, failures may surface as a generic HA "Timeout while updating data" rather than a specific connection error.
  4. Monitor Lewes/Eastbourne sensor availability during outages

    master
    For Lewes and Eastbourne councils, the integration has been updated (PR #2157) to detect EnvironmentFirst database outages. During such an outage, sensors will now transition to the unavailable state with a clear log message, rather than silently displaying stale (last known good) collection dates.
  5. Selenium and Chromium requirements

    master

    For councils that require Selenium for data fetching (indicated by a web_driver key in their configuration), the integration performs the following checks:

    1. Selenium Server Check: Verifies accessibility of remote Selenium server URLs (or a custom URL if provided).
    2. Chromium Installation Check: Verifies that a local Chromium browser is installed on the host system.

    Status results are displayed in the council-specific configuration form to assist with troubleshooting.

  6. Configure the UK Bin Collection integration

    master

    The integration configuration is a multi-step process.

    Step 1: Basic Setup

    Provide the following initial details:

    • name: A unique identifier for this configuration.
    • council: Select your council from the dropdown (listed by its wiki name).
    • manual_refresh_only (Optional): If enabled, the integration will not auto-update; you must trigger updates manually. Defaults to False.
    • icon_color_mapping (Optional): A JSON string to customize bin icons and colors.

    Step 2: Council-Specific Details

    Fields in this step are dynamic and depend on your selected council. Common fields include:

    • url: The data access URL (may be pre-filled).
    • uprn: Unique Property Reference Number.
    • postcode: Your address postcode.
    • number: Your house number.
    • usrn: Unique Street Reference Number.
    • web_driver: Command for Selenium if required by the council.
    • headless: Whether to run the browser in headless mode (default True).
    • local_browser: Whether to use a local browser instance (default False).
    • timeout: Request timeout in seconds (minimum 10, default 60).
    • update_interval: Refresh frequency in hours (minimum 1, default 12).
  7. Run UK Bin Collection Data in a terminal

    master

    To run the project manually from a terminal using Poetry, follow these steps:

    1. Clone or download the repository.
    2. Ensure you have a valid installation of Python 3.8 or higher available in your PATH.
    3. Navigate into the UKBinCollectionData directory.
    4. Install dependencies using poetry install.
    5. Enter the virtual environment with poetry shell.
    6. Change your working directory to uk_bin_collection/uk_bin_collection/.
    7. Execute the specific parser required for your local council. You can find the list of supported councils and their corresponding parsers in the Councils wiki.
    # Example sequence of commands
    git clone <repo_url>
    cd UKBinCollectionData
    poetry install
    poetry shell
    cd uk_bin_collection/uk_bin_collection/
    # Run your council's parser here
  8. Set up a local development environment for testing

    master

    To test the component against a specific Home Assistant version and run tests locally, follow these steps from the project root:

    1. Install a specific Home Assistant version via pip.
    2. Install the component in editable (development) mode.
    3. Run the compatibility checker.
    4. Execute the component's test suite using pytest.
    # Install specific HA version
    pip install homeassistant==2024.12.0
    
    # Install component in development mode
    pip install -e .
    
    # Run compatibility check
    python scripts/check_ha_compatibility.py
    
    # Run component tests
    python -m pytest custom_components/uk_bin_collection/tests/
  9. Handle Ealing Council configuration in Home Assistant

    master

    Due to a deduplication of the Ealing Council integration, EalingCouncil is now a shim that delegates to LondonBoroughEaling. To avoid configuration conflicts in the Home Assistant dropdown, the wiki_name for the old module has been updated.

    When configuring via the Home Assistant UI, look for "Ealing (deprecated, use LondonBoroughEaling)" to ensure you are using the correct, updated implementation. Existing users with the old configuration saved will remain unaffected as the module name is still importable.

  10. Automate bin calendar updates with Cron

    master

    To keep your calendar updated automatically, you can create a shell script that runs the collection and generation process, then schedule it using cron.

    Example Script (update_bin_calendar.sh):

    #!/bin/bash
    # Set variables
    COUNCIL="YourCouncilName"
    COUNCIL_URL="https://your-council-website.gov.uk/bins"
    POSTCODE="YOUR_POSTCODE"
    HOUSE_NUMBER="YOUR_HOUSE_NUMBER"
    OUTPUT_DIR="/var/www/html/calendars"
    CALENDAR_NAME="Household Bins"
    
    mkdir -p $OUTPUT_DIR
    
    cd /path/to/UKBinCollectionData && \
    python uk_bin_collection/uk_bin_collection/collect_data.py $COUNCIL "$COUNCIL_URL" \
      -p "$POSTCODE" -n "$HOUSE_NUMBER" | \
    python bin_to_ics.py --name "$CALENDAR_NAME" --output "$OUTPUT_DIR/bins.ics" --alarms "1d,6h"
    
    echo "Calendar last updated: $(date)" > "$OUTPUT_DIR/last_update.txt"

    Scheduling: To run the script every Monday at 2 AM, add this to your crontab: 0 2 * * 1 /path/to/update_bin_calendar.sh

    #!/bin/bash
    # File: update_bin_calendar.sh
    
    # Set variables
    COUNCIL="YourCouncilName"
    COUNCIL_URL="https://your-council-website.gov.uk/bins"
    POSTCODE="YOUR_POSTCODE"
    HOUSE_NUMBER="YOUR_HOUSE_NUMBER"
    OUTPUT_DIR="/var/www/html/calendars"  # Web-accessible directory
    CALENDAR_NAME="Household Bins"
    
    # Ensure output directory exists
    mkdir -p $OUTPUT_DIR
    
    # Run the collector and generate the calendar
    cd /path/to/UKBinCollectionData && \
    python uk_bin_collection/uk_bin_collection/collect_data.py $COUNCIL "$COUNCIL_URL" \
      -p "$POSTCODE" -n "$HOUSE_NUMBER" | \
    python bin_to_ics.py --name "$CALENDAR_NAME" --output "$OUTPUT_DIR/bins.ics" --alarms "1d,6h"
    
    # Add timestamp to show last update time
    echo "Calendar last updated: $(date)" > "$OUTPUT_DIR/last_update.txt"
  11. Run Selenium via Docker for Home Assistant integration

    master

    Since the official Home Assistant Selenium add-on has been removed, users needing Selenium-based scrapers should run a standalone Selenium instance using Docker.

    A common approach is to use the selenium/standalone-chrome image. This is particularly important for councils that have not yet migrated to pure HTTP/JSON scrapers.

    # Example: Running a standalone Chrome Selenium container
    docker run -d -p 4444:4444 -p 7900:7900 --shm-size="2g" selenium/standalone-chrome
  12. Set up a GitHub App for release workflows

    master

    To allow the release workflow to bypass branch protection rules and push commits/tags securely, you can set up a GitHub App. This is preferred over Personal Access Tokens because it provides fine-grained permissions, does not expire, and has a better audit trail.

    Step 1: Create the GitHub App

    1. Navigate to settings:
      • Personal account: https://github.com/settings/apps/new
      • Organization: https://github.com/organizations/YOUR_ORG/settings/apps/new
    2. Basic Info:
      • GitHub App name: UKBinCollection Release Bot (or a unique variation like UKBinCollection-Release-Bot-YourUsername)
      • Homepage URL: https://github.com/robbrad/UKBinCollectionData
    3. Webhook: Uncheck "Active" under "Webhook".
    4. Permissions:
      • Contents: Read and write (Required for pushing commits and tags)
      • Metadata: Read-only (Default)
      • Pull requests: Read and write (Optional)
    5. Installation Scope: Select "Only on this account".
    6. Click "Create GitHub App".

    Step 2: Install the App

    1. In the app settings, click "Install App" in the left sidebar.
    2. Click "Install" next to your account/organization.
    3. Select "Only select repositories" and check UKBinCollectionData.
    4. Click "Install".

    Step 3: Generate and Retrieve Credentials

    1. Generate Private Key: In the app settings, scroll to "Private keys" and click "Generate a private key". A .pem file will download.
    2. App ID: Copy the numeric ID found at the top of the app settings page.
    3. Private Key Content: Open the .pem file and copy the entire contents, including the headers:
      -----BEGIN RSA PRIVATE KEY-----
      [long string of characters]
      -----END RSA PRIVATE KEY-----

    Step 4: Add Secrets to Repository

    Navigate to your repository's Settings → Secrets and variables → Actions and add the following:

    Secret NameValueNote
    APP_IDYour App ID (e.g., 123456)Just the number, no quotes
    APP_PRIVATE_KEYThe full .pem contentMust include BEGIN and END lines

    Step 5: Verify

    Merge a PR with a conventional commit message to trigger the bump workflow and check the logs.

    -----BEGIN RSA PRIVATE KEY-----
    [long string of characters]
    -----END RSA PRIVATE KEY-----