Watchtower

repository·main·Indexed 12 days ago

https://github.com/containrrr/watchtower

A tool for automating the update process of Docker containers by monitoring image registries for new versions and automatically restarting containers with the updated images. It supports daemon mode with cron scheduling, run-once execution, and an HTTP API for triggering updates. Key features include image and volume cleanup, container dependency management via labels, and integration with shoutrrr for notifications.

Tokens
18K
Snippets
59
Records
78
Agent score
95%

What's inside Watchtower

  1. What is Watchtower

    main
    Watchtower is a process designed to automate Docker container base image updates. It monitors your running containers, pulls down new images when they are pushed to Docker Hub or your own registry, gracefully shuts down the existing container, and restarts it using the same configuration options used during its initial deployment.
  2. What is Watchtower and how does it work?

    main

    Watchtower is an application that monitors running Docker containers for changes to their original images. When it detects that a new image is available in your registry (e.g., Docker Hub), it automatically performs the following lifecycle:

    1. Pulls the new image.
    2. Gracefully shuts down the existing container.
    3. Restarts the container using the new image while preserving the exact same configuration and docker run options (such as port mappings, environment variables, and volumes) used during the initial deployment.
  3. Filter containers by scope

    main

    You can run multiple instances of Watchtower on the same Docker daemon by using scopes. This prevents instances from interfering with each other's containers.

    Containers must have the label com.centurylinklabs.watchtower.scope set to a specific value. Use the --scope argument or WATCHTOWER_SCOPE environment variable to match that value.

    Note: If you want other Watchtower instances to ignore scoped containers, set the scope argument to none.

    # Example: Watchtower instance only managing containers with scope 'production'
    watchtower --scope production
  4. How Watchtower container filtering works

    main

    Watchtower determines which containers to monitor by testing them against all configured criteria. A container is only monitored if it meets all criteria.

    Common filtering interactions include:

    • Name filtering vs Labels: If you provide a specific list of names via the --name argument, a container will not be monitored if its name is not in that list, even if it has com.centurylinklabs.watchtower.enable=true and you are using the --label-enable flag.
    • Name filtering vs Enable Label: If a container's name is in the monitoring name list but it has com.centurylinklabs.watchtower.enable=false, it will not be monitored.
  5. Understand the notification template structure

    main

    Watchtower uses Go templates to format notification messages. The template engine provides access to two primary data structures: .Report and .Entries.

    .Report Object

    Used to summarize the status of container updates. It contains several collections that can be iterated over:

    • .Scanned: Containers that were checked.
    • .Updated: Containers that were successfully updated.
    • .Failed: Containers that failed to update.
    • .Skipped: Containers that were bypassed.
    • .Fresh: Containers that were already up to date.
    • .Stale: Containers that require an update.

    Each item in these collections typically provides access to:

    • .Name: The container name.
    • .ImageName: The name of the image.
    • .State: The current state of the container.
    • .Error: Error messages if applicable.
    • .CurrentImageID.ShortID and .LatestImageID.ShortID: Shortened image identifiers for comparison.

    .Entries Object

    Used to include log entries in the notification. Each entry in the .Entries slice contains:

    • .Time: The timestamp of the log (can be formatted using Go's time formatting, e.g., .Time.Format "2006-01-02T15:04:05Z07:00").
    • .Level: The log level (e.g., error, warning, info, debug).
    • .Message: The actual log message text.
  6. Implicit links with network_mode: service:container

    main
    Watchtower treats containers using network_mode: service:container as having an implicit link. If a container is configured to use the network namespace of another service, Watchtower will automatically manage the dependency order for those containers during updates.
  7. Use Simple vs Report templates for notifications

    main

    Watchtower supports two modes of notification messaging based on the template data provided:

    Simple Templates

    By default, Watchtower uses Simple templates. These format a list of logrus log entries. The default template is {{range .}}{{.Message}}{{println}}{{end}}.

    To avoid sending empty notifications, wrap your template with {{if .}}{{.}}{{end}}.

    Report Templates

    If you specify the --notification-report (env WATCHTOWER_NOTIFICATION_REPORT) flag, Watchtower switches to Report templates. These use a notification.Data struct to provide a summary of the entire session (e.g., how many containers were scanned, updated, failed, or skipped).

    Default Report Template Structure:

    {{- if .Report -}}
      {{- with .Report -}}
        {{- if ( or .Updated .Failed ) -}}
    {{len .Scanned}} Scanned, {{len .Updated}} Updated, {{len .Failed}} Failed
          {{- range .Updated}}
    - {{.Name}} ({{.ImageName}}): {{.CurrentImageID.ShortID}} updated to {{.LatestImageID.ShortID}}
          {{- end -}}
          {{- range .Fresh}}
    - {{.Name}} ({{.ImageName}}): {{.State}}
    	  {{- end -}}
          {{- range .Skipped}}
    - {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}}
    	  {{- end -}}
          {{- range .Failed}}
    - {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}}
          {{- end -}}
        {{- end -}}
      {{- end -}}
    {{- else -}}
      {{range .Entries -}}{{.Message}}{{"\n"}}{{- end -}}
    {{- end -}}
  8. How Watchtower handles linked containers

    main

    Watchtower automatically detects links between running containers to ensure updates do not break application dependencies. When an update is detected for a container that has dependencies, Watchtower manages the shutdown and startup sequence to maintain link integrity.

    Shutdown Order: Watchtower shuts down dependent containers first, then the dependency itself. Startup Order: Watchtower starts the dependency first, then the dependent container.

    Example Scenario: If a wordpress container is linked to a mysql container and mysql requires an update:

    1. Watchtower shuts down wordpress.
    2. Watchtower shuts down mysql.
    3. Watchtower starts mysql.
    4. Watchtower starts wordpress.
  9. Authenticate with private Docker registries using config.json

    main

    Watchtower can access private registries by using a config.json file containing credentials. You can either create this file manually or share an existing Docker configuration file generated via docker login.

    Manual Configuration

    Create a config.json file with the following structure. The auth value must be a base64 encoded string of username:password.

    {
        "auths": {
            "<REGISTRY_NAME>": {
                "auth": "XXXXXXX"
            }
        }
    }

    Registry Name Rules:

    • Docker Hub: Use https://index.docker.io/v1/. Watchtower will use these credentials for any image without a specified registry domain.
    • Local Registries: Ensure the host matches exactly in both config.json and your container definitions (e.g., localhost, 127.0.0.1, or host.domain:port).
    • GCloud: Use _json_key as the username and the content of your gcloudauth.json as the password.

    Using an existing Docker config

    If you have already run docker login, you can mount your existing ~/.docker/config.json directly into Watchtower.

    Mounting the config file

    Pass the configuration file to the Watchtower container using a volume mount. By default, Watchtower looks for the file at /config.json inside the container.

    Docker CLI:

    docker run [...] -v <PATH>/config.json:/config.json containrrr/watchtower

    Docker Compose:

    version: "3.4"
    services:
      watchtower:
        image: containrrr/watchtower:latest
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - <PATH_TO_HOME_DIR>/.docker/config.json:/config.json
    # Generate base64 auth string
    echo -n 'username:password' | base64
    
    # For GCloud
    echo -n "_json_key:$(cat gcloudauth.json)" | base64 -w0
  10. Enable lifecycle hooks in Watchtower

    main

    Lifecycle hooks allow you to execute shell commands inside a container before or after an update. This feature is disabled by default. To enable it, use one of the following methods:

    • CLI Flag: Add --enable-lifecycle-hooks to your Watchtower command.
    • Environment Variable: Set WATCHTOWER_LIFECYCLE_HOOKS to true.

    Important Requirements & Constraints:

    • Shell Availability: Commands are executed using sh, so the target container must include the sh executable.
    • Container State: If a container is not currently running, lifecycle hooks will not execute, and the update will proceed without them.
    • Failure Behavior: If a command fails (returns an exit code other than 0 or 75), Watchtower will log the error but will not stop the update process.
    # Using environment variable
    WATCHTOWER_LIFECYCLE_HOOKS=true watchtower
    
    # Using CLI flag
    watchtower --enable-lifecycle-hooks
  11. Configure Watchtower notifications via Shoutrrr

    main

    Watchtower uses the Shoutrrr library to send notifications. You can specify one or multiple notification services by providing a space-separated list of Shoutrrr service URLs.

    Key Configuration Options

    • --notification-url (env WATCHTOWER_NOTIFICATION_URL): The Shoutrrr service URL(s). This can also be a path to a file containing the URL(s).
    • --notification-template (env WATCHTOWER_NOTIFICATION_TEMPLATE): A Go text/template used to format the message.
    • --notification-report (env WATCHTOWER_NOTIFICATION_REPORT): When enabled, Watchtower uses the session report data instead of individual log entries for the template.

    Important: Multiple Notifications Workaround

    Due to a bug in Viper, you cannot use comma-separated values in environment variables for multiple notifications. Instead, use spaces to separate URLs and wrap the value in quotes:

    WATCHTOWER_NOTIFICATION_URL="slack://token1 slack://token2"

    Note: If using docker-compose, do not use double quotes inside your .yml file for these values to avoid startup errors.

    docker run -d \
      --name watchtower \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -e WATCHTOWER_NOTIFICATION_URL="discord://token@channel slack://watchtower@token-a/token-b/token-c" \
      -e WATCHTOWER_NOTIFICATION_TEMPLATE="{{range .}}{{.Time.Format \"2006-01-02 15:04:05\"}} ({{.Level}}): {{.Message}}{{println}}{{end}}" \
      containrrr/watchtower
  12. Configure custom stop signals for containers

    main

    By default, Watchtower sends a SIGTERM signal to stop a container when an update is detected. If your application requires a different signal to shut down gracefully, you can specify it using the label com.centurylinklabs.watchtower.stop-signal.

    You can apply this label either during the image build process in your Dockerfile or when starting the container via the docker run command.

    ### Using a Dockerfile
    ```docker
    LABEL com.centurylinklabs.watchtower.stop-signal="SIGHUP"

    Using docker run

    docker run -d --label=com.centurylinklabs.watchtower.stop-signal=SIGHUP someimage