nebula-sync

repository·main·Indexed 23 days ago

https://github.com/lovelaze/nebula-sync

A tool to synchronize Pi-hole v6.x configurations from a primary instance to one or more replica instances. It supports full synchronization via Pi-hole Teleporter as well as selective synchronization of specific configuration components, including DNS, DHCP, NTP, and gravity lists. It can be deployed as a Linux/OSX binary or via Docker, and supports scheduled synchronization via cron and outcome notifications via webhooks.

Tokens
3K
Snippets
7
Records
19
Agent score
76%

What's inside nebula-sync

  1. Deploy nebula-sync with Docker CLI

    main

    Run nebula-sync using a single Docker command by passing configuration through -e flags.

    docker run --rm \
      --name nebula-sync \
      -e PRIMARY="http://ph1.example.com|password" \
      -e REPLICAS="http://ph2.example.com|password" \
      -e FULL_SYNC=true \
      -e RUN_GRAVITY=true \
      ghcr.io/lovelaze/nebula-sync:latest
  2. Install nebula-sync

    main

    You can install nebula-sync as a Linux/OSX binary or via Docker.

    To build from source using Go:

    go install github.com/lovelaze/nebula-sync@latest

    To run the binary directly:

    # Standard run
    nebula-sync run
    
    # Run using environment variables from a specific file
    nebula-sync run --env-file .env
  3. Deploy nebula-sync with Docker Compose

    main

    Docker Compose is the recommended method for deployment. Use the ghcr.io/lovelaze/nebula-sync:latest image and provide configuration via environment variables.

    Example docker-compose.yml:

    services:
      nebula-sync:
        image: ghcr.io/lovelaze/nebula-sync:latest
        container_name: nebula-sync
        environment:
        - PRIMARY=http://ph1.example.com|password
        - REPLICAS=http://ph2.example.com|password,http://ph3.example.com|password
        - FULL_SYNC=true
        - RUN_GRAVITY=true
        - CRON=0 * * * *
  4. Filter configuration keys during selective sync

    main

    When FULL_SYNC=false, you can include or exclude specific configuration keys within a section.

    Rules:

    • SYNC_CONFIG_<SECTION>_INCLUDE and SYNC_CONFIG_<SECTION>_EXCLUDE are mutually exclusive within a section.
    • Keys are case sensitive and relative to the section. (e.g., for dns.upstreams, use upstreams).

    Available Filters:

    NameExampleDescription
    SYNC_CONFIG_DNS_INCLUDEupstreams,interfaceDNS config keys to include
    SYNC_CONFIG_DNS_EXCLUDEupstreams,interfaceDNS config keys to exclude
    SYNC_CONFIG_DHCP_INCLUDEactive,startDHCP config keys to include
    SYNC_CONFIG_DHCP_EXCLUDEactive,startDHCP config keys to exclude
    SYNC_CONFIG_NTP_INCLUDEipv4,syncNTP config keys to include
    SYNC_CONFIG_NTP_EXCLUDEipv4,syncNTP config keys to exclude
    SYNC_CONFIG_RESOLVER_INCLUDEresolveIPv4,networkNamesResolver config keys to include
    SYNC_CONFIG_RESOLVER_EXCLUDEresolveIPv4,networkNamesResolver config keys to exclude
    SYNC_CONFIG_DATABASE_INCLUDEDBimport,maxDBdaysDatabase config keys to include
    SYNC_CONFIG_DATABASE_EXCLUDEDBimport,maxDBdaysDatabase config keys to exclude
    SYNC_CONFIG_MISC_INCLUDEnice,delay_startupMisc config keys to include
    SYNC_CONFIG_MISC_EXCLUDEnice,delay_startupMisc config keys to exclude
    SYNC_CONFIG_DEBUG_INCLUDEdatabase,networkingDebug config keys to include
    SYNC_CONFIG_DEBUG_EXCLUDEdatabase,networkingDebug config keys to exclude
  5. Configure Required Environment Variables

    main

    The following environment variables must be provided for nebula-sync to function:

    NameDescription
    PRIMARYSpecifies the primary Pi-hole configuration (Format: http://host|password)
    REPLICASA comma-separated list of replica Pi-hole configurations (Format: http://host|password,http://host|password)
    FULL_SYNCBoolean (true/false). If true, performs a full Teleporter import/export from primary to replicas.

    Note: For Docker users, PRIMARY and REPLICAS can be provided via Docker secrets using PRIMARY_FILE and REPLICAS_FILE respectively.

  6. Configure Optional Environment Variables

    main

    Use these variables to control scheduling, client behavior, and synchronization granularity.

    General Settings

    NameDefaultDescription
    CRONn/aCron schedule for synchronization (e.g., 0 * * * *)
    RUN_GRAVITYfalseWhether to run gravity after syncing
    TZEurope/LondonTimezone for logs and cron
    CLIENT_SKIP_TLS_VERIFICATIONfalseSkips TLS certificate verification
    CLIENT_RETRY_DELAY_SECONDS1Seconds to delay between connection attempts
    CLIENT_TIMEOUT_SECONDS20HTTP client timeout in seconds

    Selective Sync Settings (Only applicable if FULL_SYNC=false)

    These allow granular control over which parts of the Pi-hole configuration are synchronized.

    NameDefaultDescription
    SYNC_CONFIG_DNSfalseSynchronize DNS settings
    SYNC_CONFIG_DHCPfalseSynchronize DHCP settings
    SYNC_CONFIG_NTPfalseSynchronize NTP settings
    SYNC_CONFIG_RESOLVERfalseSynchronize resolver settings
    SYNC_CONFIG_DATABASEfalseSynchronize database settings
    SYNC_CONFIG_MISCfalseSynchronize miscellaneous settings
    SYNC_CONFIG_DEBUGfalseSynchronize debug settings
    SYNC_GRAVITY_DHCP_LEASESfalseSynchronize DHCP leases
    SYNC_GRAVITY_GROUPfalseSynchronize groups
    SYNC_GRAVITY_AD_LISTfalseSynchronize ad lists
    SYNC_GRAVITY_AD_LIST_BY_GROUPfalseSynchronize ad lists by group
    SYNC_GRAVITY_DOMAIN_LISTfalseSynchronize domain lists
    SYNC_GRAVITY_DOMAIN_LIST_BY_GROUPfalseSynchronize domain lists by group
    SYNC_GRAVITY_CLIENTfalseSynchronize clients
    SYNC_GRAVITY_CLIENT_BY_GROUPfalseSynchronize clients by group
  7. Configure Webhooks for sync outcomes

    main

    Nebula Sync can trigger webhooks on SUCCESS or FAILURE. Webhooks have a 10-second timeout. Replace <OUTCOME> in the variable names with either SUCCESS or FAILURE.

    NameDefaultDescription
    WEBHOOK_SYNC_<OUTCOME>_URLn/aURL to invoke
    WEBHOOK_SYNC_<OUTCOME>_METHODPOSTHTTP method
    WEBHOOK_SYNC_<OUTCOME>_BODYn/aRequest body
    WEBHOOK_SYNC_<OUTCOME>_HEADERSn/aHeaders in key:value,key2:value2 format (whitespace is preserved)
    WEBHOOK_CLIENT_SKIP_TLS_VERIFICATIONfalseSkips TLS verification for all webhooks

    Webhook Examples

    Healthcheck.io:

    WEBHOOK_SYNC_SUCCESS_URL=https://hc-ping.com/{your-slug-or-guid-here}
    WEBHOOK_SYNC_FAILURE_URL=https://hc-ping.com/{your-slug-or-guid-here}/fail

    Apprise (Notification):

    WEBHOOK_SYNC_FAILURE_URL=http://localhost:8080/notify
    WEBHOOK_SYNC_FAILURE_BODY=urls=mailto://user:pass@gmail.com&body=test message

    JSON Service:

    WEBHOOK_SYNC_FAILURE_URL=https://www.example.com/notify.json
    WEBHOOK_SYNC_FAILURE_BODY={"hello":"world"}
    WEBHOOK_SYNC_FAILURE_HEADERS=Content-Type:application/json
  8. Resolve Pi-hole authentication errors with app passwords

    main

    When using Pi-hole's app passwords with nebula-sync, you must enable the webserver.api.app_sudo setting on all REPLICAS servers to avoid authentication errors. You can configure this in one of three ways:

    Method 1: Command Line

    Execute the following command on each replica:

    sudo pihole-FTL --config webserver.api.app_sudo true

    Method 2: Pi-hole Web UI

    1. Navigate to Settings -> All Settings.
    2. Toggle the Modified settings / All settings slider in the upper right to show All settings.
    3. Locate the Webserver and API section.
    4. Check the Enabled box under webserver.api.app_sudo.
    5. Click Save & Apply.
    6. Repeat for each replica.

    Method 3: Configuration File

    1. Edit /etc/pihole/pihole.toml on each replica.
    2. Find the line app_sudo = false under the [webserver.api] section.
    3. Change it to app_sudo = true.
    4. Save the file and restart the pihole-FTL service.
  9. Fix Docker permission denied errors for secrets

    main

    By default, the nebula-sync Docker container runs as user 1001. If you are using Docker secrets, the container user must have read permissions for the secret files. If you encounter the error Failed to initialize service error="open /run/secrets/primary: permission denied", you must resolve the permission mismatch using one of these two methods:

    1. Adjust file permissions: Ensure the secret files are owned by user 1001 and have appropriate read permissions:
      chown 1001 ./your/secretfiles && chmod 400 ./your/secretfiles
    2. Change the container user: Use the user directive in your Docker Compose file to run the container as a different user, then ensure your secret files are owned by that specific user.
    chown 1001 ./your/secretfiles && chmod 400 ./your/secretfiles
  10. Initialize a Pi-hole Client with NewClient

    main
    To interact with a Pi-hole instance, use NewClient. It requires a model.PiHole configuration object (containing the URL and password) and an *http.Client for making requests. The returned Client interface provides methods for authentication, configuration management, and Teleporter operations.
  11. Manage Pi-hole Configuration with GetConfig and PatchConfig

    main

    You can retrieve and update the Pi-hole configuration using the following methods:

    • GetConfig(): Returns a *model.ConfigResponse containing the current configuration.
    • PatchConfig(patchRequest *model.PatchConfigRequest): Sends a patch request to update specific configuration settings.

    Both methods require a valid authenticated session.

  12. Use Teleporter to backup or restore Pi-hole settings

    main

    Teleporter is used for full configuration backups and restores. The Client interface provides:

    • GetTeleporter() ([]byte, error): Downloads the current Teleporter backup as a byte slice (typically a .zip file).
    • PostTeleporter(payload []byte, teleporterRequest *model.PostTeleporterRequest) error: Uploads a new Teleporter backup. The payload should be the raw bytes of the backup file (e.g., config.zip). You can optionally provide a *model.PostTeleporterRequest to include import metadata via a multipart form field named import.