UK Bin Collection Data (UKBCD)
repository·master·Indexed 43 days ago
https://github.com/robbrad/ukbincollectiondataA Python library and Home Assistant integration (version 0.172.3) that provides a standardized way to access UK council bin collection schedules in JSON format. It supports various scraping methods, including Beautiful Soup 4 and Selenium, to extract data from council websites for use in home automation.
What's inside UK Bin Collection Data
- UK Bin Collection Data (UKBCD) is a project that provides UK council bin collection data in a standardized JSON format. It is designed for use in home automation (like Home Assistant) to trigger actions, such as lighting up an LED bar on bin collection days. The project uses various scraping methods, including Beautiful Soup 4, to extract data from council websites, CSV files, or by emulating web browser behavior.
Understand the manual_refresh_only configuration behavior
masterThe configuration key
manual_refresh_onlycan be confusing due to its naming.- UI Label: "Automatically refresh the sensor"
- Behavior: When this flag is set, it controls whether the sensor updates automatically.
Despite the name
manual_refresh_only, the logic is designed to match the Home Assistant UI label. Do not attempt to invert the logic to match the key name, as this will break the expected 'automatic refresh' behavior for existing users.Understand the impact of shared HTTP fetch retries
masterThe project implements a shared HTTP fetch retry mechanism (via PR #2155) for councils using the default
requests-based fetch path (approximately 101 of the 352 councils).Side Effects
- Resilience: Increases stability against transient network errors.
- Increased Failure Latency: If a council's website is genuinely down, the integration will take longer to report a failure (up to ~46s of retry backoff plus connection time) instead of failing immediately.
- Error Reporting: Because retries can outlast Home Assistant's per-update timeout, failures may surface as a generic HA
"Timeout while updating data"rather than a specific connection error.
Monitor Lewes/Eastbourne sensor availability during outages
masterFor Lewes and Eastbourne councils, the integration has been updated (PR #2157) to detect EnvironmentFirst database outages. During such an outage, sensors will now transition to theunavailablestate with a clear log message, rather than silently displaying stale (last known good) collection dates.Selenium and Chromium requirements
masterFor councils that require Selenium for data fetching (indicated by a
web_driverkey in their configuration), the integration performs the following checks:- Selenium Server Check: Verifies accessibility of remote Selenium server URLs (or a custom URL if provided).
- Chromium Installation Check: Verifies that a local Chromium browser is installed on the host system.
Status results are displayed in the council-specific configuration form to assist with troubleshooting.
Configure the UK Bin Collection integration
masterThe integration configuration is a multi-step process.
Step 1: Basic Setup
Provide the following initial details:
name: A unique identifier for this configuration.council: Select your council from the dropdown (listed by its wiki name).manual_refresh_only(Optional): If enabled, the integration will not auto-update; you must trigger updates manually. Defaults toFalse.icon_color_mapping(Optional): A JSON string to customize bin icons and colors.
Step 2: Council-Specific Details
Fields in this step are dynamic and depend on your selected council. Common fields include:
url: The data access URL (may be pre-filled).uprn: Unique Property Reference Number.postcode: Your address postcode.number: Your house number.usrn: Unique Street Reference Number.web_driver: Command for Selenium if required by the council.headless: Whether to run the browser in headless mode (defaultTrue).local_browser: Whether to use a local browser instance (defaultFalse).timeout: Request timeout in seconds (minimum10, default60).update_interval: Refresh frequency in hours (minimum1, default12).
Run UK Bin Collection Data in a terminal
masterTo run the project manually from a terminal using Poetry, follow these steps:
- Clone or download the repository.
- Ensure you have a valid installation of Python 3.8 or higher available in your PATH.
- Navigate into the
UKBinCollectionDatadirectory. - Install dependencies using
poetry install. - Enter the virtual environment with
poetry shell. - Change your working directory to
uk_bin_collection/uk_bin_collection/. - Execute the specific parser required for your local council. You can find the list of supported councils and their corresponding parsers in the Councils wiki.
# Example sequence of commands git clone <repo_url> cd UKBinCollectionData poetry install poetry shell cd uk_bin_collection/uk_bin_collection/ # Run your council's parser hereSet up a local development environment for testing
masterTo test the component against a specific Home Assistant version and run tests locally, follow these steps from the project root:
- Install a specific Home Assistant version via pip.
- Install the component in editable (development) mode.
- Run the compatibility checker.
- Execute the component's test suite using
pytest.
# Install specific HA version pip install homeassistant==2024.12.0 # Install component in development mode pip install -e . # Run compatibility check python scripts/check_ha_compatibility.py # Run component tests python -m pytest custom_components/uk_bin_collection/tests/Handle Ealing Council configuration in Home Assistant
masterDue to a deduplication of the Ealing Council integration,
EalingCouncilis now a shim that delegates toLondonBoroughEaling. To avoid configuration conflicts in the Home Assistant dropdown, thewiki_namefor the old module has been updated.When configuring via the Home Assistant UI, look for "Ealing (deprecated, use LondonBoroughEaling)" to ensure you are using the correct, updated implementation. Existing users with the old configuration saved will remain unaffected as the module name is still importable.
Automate bin calendar updates with Cron
masterTo keep your calendar updated automatically, you can create a shell script that runs the collection and generation process, then schedule it using
cron.Example Script (
update_bin_calendar.sh):#!/bin/bash # Set variables COUNCIL="YourCouncilName" COUNCIL_URL="https://your-council-website.gov.uk/bins" POSTCODE="YOUR_POSTCODE" HOUSE_NUMBER="YOUR_HOUSE_NUMBER" OUTPUT_DIR="/var/www/html/calendars" CALENDAR_NAME="Household Bins" mkdir -p $OUTPUT_DIR cd /path/to/UKBinCollectionData && \ python uk_bin_collection/uk_bin_collection/collect_data.py $COUNCIL "$COUNCIL_URL" \ -p "$POSTCODE" -n "$HOUSE_NUMBER" | \ python bin_to_ics.py --name "$CALENDAR_NAME" --output "$OUTPUT_DIR/bins.ics" --alarms "1d,6h" echo "Calendar last updated: $(date)" > "$OUTPUT_DIR/last_update.txt"Scheduling: To run the script every Monday at 2 AM, add this to your crontab:
0 2 * * 1 /path/to/update_bin_calendar.sh#!/bin/bash # File: update_bin_calendar.sh # Set variables COUNCIL="YourCouncilName" COUNCIL_URL="https://your-council-website.gov.uk/bins" POSTCODE="YOUR_POSTCODE" HOUSE_NUMBER="YOUR_HOUSE_NUMBER" OUTPUT_DIR="/var/www/html/calendars" # Web-accessible directory CALENDAR_NAME="Household Bins" # Ensure output directory exists mkdir -p $OUTPUT_DIR # Run the collector and generate the calendar cd /path/to/UKBinCollectionData && \ python uk_bin_collection/uk_bin_collection/collect_data.py $COUNCIL "$COUNCIL_URL" \ -p "$POSTCODE" -n "$HOUSE_NUMBER" | \ python bin_to_ics.py --name "$CALENDAR_NAME" --output "$OUTPUT_DIR/bins.ics" --alarms "1d,6h" # Add timestamp to show last update time echo "Calendar last updated: $(date)" > "$OUTPUT_DIR/last_update.txt"Run Selenium via Docker for Home Assistant integration
masterSince the official Home Assistant Selenium add-on has been removed, users needing Selenium-based scrapers should run a standalone Selenium instance using Docker.
A common approach is to use the
selenium/standalone-chromeimage. This is particularly important for councils that have not yet migrated to pure HTTP/JSON scrapers.# Example: Running a standalone Chrome Selenium container docker run -d -p 4444:4444 -p 7900:7900 --shm-size="2g" selenium/standalone-chromeSet up a GitHub App for release workflows
masterTo allow the release workflow to bypass branch protection rules and push commits/tags securely, you can set up a GitHub App. This is preferred over Personal Access Tokens because it provides fine-grained permissions, does not expire, and has a better audit trail.
Step 1: Create the GitHub App
- Navigate to settings:
- Personal account:
https://github.com/settings/apps/new - Organization:
https://github.com/organizations/YOUR_ORG/settings/apps/new
- Personal account:
- Basic Info:
- GitHub App name:
UKBinCollection Release Bot(or a unique variation likeUKBinCollection-Release-Bot-YourUsername) - Homepage URL:
https://github.com/robbrad/UKBinCollectionData
- GitHub App name:
- Webhook: Uncheck "Active" under "Webhook".
- Permissions:
- Contents:
Read and write(Required for pushing commits and tags) - Metadata:
Read-only(Default) - Pull requests:
Read and write(Optional)
- Contents:
- Installation Scope: Select "Only on this account".
- Click "Create GitHub App".
Step 2: Install the App
- In the app settings, click "Install App" in the left sidebar.
- Click "Install" next to your account/organization.
- Select "Only select repositories" and check
UKBinCollectionData. - Click "Install".
Step 3: Generate and Retrieve Credentials
- Generate Private Key: In the app settings, scroll to "Private keys" and click "Generate a private key". A
.pemfile will download. - App ID: Copy the numeric ID found at the top of the app settings page.
- Private Key Content: Open the
.pemfile and copy the entire contents, including the headers:-----BEGIN RSA PRIVATE KEY----- [long string of characters] -----END RSA PRIVATE KEY-----
Step 4: Add Secrets to Repository
Navigate to your repository's Settings → Secrets and variables → Actions and add the following:
Secret Name Value Note APP_IDYour App ID (e.g., 123456)Just the number, no quotes APP_PRIVATE_KEYThe full .pemcontentMust include BEGINandENDlinesStep 5: Verify
Merge a PR with a conventional commit message to trigger the bump workflow and check the logs.
-----BEGIN RSA PRIVATE KEY----- [long string of characters] -----END RSA PRIVATE KEY------ Navigate to settings: