Watchtower
repository·main·Indexed 12 days ago
https://github.com/containrrr/watchtowerA tool for automating the update process of Docker containers by monitoring image registries for new versions and automatically restarting containers with the updated images. It supports daemon mode with cron scheduling, run-once execution, and an HTTP API for triggering updates. Key features include image and volume cleanup, container dependency management via labels, and integration with shoutrrr for notifications.
What's inside Watchtower
- Watchtower is a process designed to automate Docker container base image updates. It monitors your running containers, pulls down new images when they are pushed to Docker Hub or your own registry, gracefully shuts down the existing container, and restarts it using the same configuration options used during its initial deployment.
What is Watchtower and how does it work?
mainWatchtower is an application that monitors running Docker containers for changes to their original images. When it detects that a new image is available in your registry (e.g., Docker Hub), it automatically performs the following lifecycle:
- Pulls the new image.
- Gracefully shuts down the existing container.
- Restarts the container using the new image while preserving the exact same configuration and
docker runoptions (such as port mappings, environment variables, and volumes) used during the initial deployment.
Filter containers by scope
mainYou can run multiple instances of Watchtower on the same Docker daemon by using scopes. This prevents instances from interfering with each other's containers.
Containers must have the label
com.centurylinklabs.watchtower.scopeset to a specific value. Use the--scopeargument orWATCHTOWER_SCOPEenvironment variable to match that value.Note: If you want other Watchtower instances to ignore scoped containers, set the scope argument to
none.# Example: Watchtower instance only managing containers with scope 'production' watchtower --scope productionHow Watchtower container filtering works
mainWatchtower determines which containers to monitor by testing them against all configured criteria. A container is only monitored if it meets all criteria.
Common filtering interactions include:
- Name filtering vs Labels: If you provide a specific list of names via the
--nameargument, a container will not be monitored if its name is not in that list, even if it hascom.centurylinklabs.watchtower.enable=trueand you are using the--label-enableflag. - Name filtering vs Enable Label: If a container's name is in the monitoring name list but it has
com.centurylinklabs.watchtower.enable=false, it will not be monitored.
- Name filtering vs Labels: If you provide a specific list of names via the
Understand the notification template structure
mainWatchtower uses Go templates to format notification messages. The template engine provides access to two primary data structures:
.Reportand.Entries..ReportObjectUsed to summarize the status of container updates. It contains several collections that can be iterated over:
.Scanned: Containers that were checked..Updated: Containers that were successfully updated..Failed: Containers that failed to update..Skipped: Containers that were bypassed..Fresh: Containers that were already up to date..Stale: Containers that require an update.
Each item in these collections typically provides access to:
.Name: The container name..ImageName: The name of the image..State: The current state of the container..Error: Error messages if applicable..CurrentImageID.ShortIDand.LatestImageID.ShortID: Shortened image identifiers for comparison.
.EntriesObjectUsed to include log entries in the notification. Each entry in the
.Entriesslice contains:.Time: The timestamp of the log (can be formatted using Go's time formatting, e.g.,.Time.Format "2006-01-02T15:04:05Z07:00")..Level: The log level (e.g.,error,warning,info,debug)..Message: The actual log message text.
Implicit links with network_mode: service:container
mainWatchtower treats containers usingnetwork_mode: service:containeras having an implicit link. If a container is configured to use the network namespace of another service, Watchtower will automatically manage the dependency order for those containers during updates.Use Simple vs Report templates for notifications
mainWatchtower supports two modes of notification messaging based on the template data provided:
Simple Templates
By default, Watchtower uses Simple templates. These format a list of logrus log entries. The default template is
{{range .}}{{.Message}}{{println}}{{end}}.To avoid sending empty notifications, wrap your template with
{{if .}}{{.}}{{end}}.Report Templates
If you specify the
--notification-report(envWATCHTOWER_NOTIFICATION_REPORT) flag, Watchtower switches to Report templates. These use anotification.Datastruct to provide a summary of the entire session (e.g., how many containers were scanned, updated, failed, or skipped).Default Report Template Structure:
{{- if .Report -}} {{- with .Report -}} {{- if ( or .Updated .Failed ) -}} {{len .Scanned}} Scanned, {{len .Updated}} Updated, {{len .Failed}} Failed {{- range .Updated}} - {{.Name}} ({{.ImageName}}): {{.CurrentImageID.ShortID}} updated to {{.LatestImageID.ShortID}} {{- end -}} {{- range .Fresh}} - {{.Name}} ({{.ImageName}}): {{.State}} {{- end -}} {{- range .Skipped}} - {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}} {{- end -}} {{- range .Failed}} - {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}} {{- end -}} {{- end -}} {{- end -}} {{- else -}} {{range .Entries -}}{{.Message}}{{"\n"}}{{- end -}} {{- end -}}How Watchtower handles linked containers
mainWatchtower automatically detects links between running containers to ensure updates do not break application dependencies. When an update is detected for a container that has dependencies, Watchtower manages the shutdown and startup sequence to maintain link integrity.
Shutdown Order: Watchtower shuts down dependent containers first, then the dependency itself. Startup Order: Watchtower starts the dependency first, then the dependent container.
Example Scenario: If a
wordpresscontainer is linked to amysqlcontainer andmysqlrequires an update:- Watchtower shuts down
wordpress. - Watchtower shuts down
mysql. - Watchtower starts
mysql. - Watchtower starts
wordpress.
- Watchtower shuts down
Authenticate with private Docker registries using config.json
mainWatchtower can access private registries by using a
config.jsonfile containing credentials. You can either create this file manually or share an existing Docker configuration file generated viadocker login.Manual Configuration
Create a
config.jsonfile with the following structure. Theauthvalue must be a base64 encoded string ofusername:password.{ "auths": { "<REGISTRY_NAME>": { "auth": "XXXXXXX" } } }Registry Name Rules:
- Docker Hub: Use
https://index.docker.io/v1/. Watchtower will use these credentials for any image without a specified registry domain. - Local Registries: Ensure the host matches exactly in both
config.jsonand your container definitions (e.g.,localhost,127.0.0.1, orhost.domain:port). - GCloud: Use
_json_keyas the username and the content of yourgcloudauth.jsonas the password.
Using an existing Docker config
If you have already run
docker login, you can mount your existing~/.docker/config.jsondirectly into Watchtower.Mounting the config file
Pass the configuration file to the Watchtower container using a volume mount. By default, Watchtower looks for the file at
/config.jsoninside the container.Docker CLI:
docker run [...] -v <PATH>/config.json:/config.json containrrr/watchtowerDocker Compose:
version: "3.4" services: watchtower: image: containrrr/watchtower:latest volumes: - /var/run/docker.sock:/var/run/docker.sock - <PATH_TO_HOME_DIR>/.docker/config.json:/config.json# Generate base64 auth string echo -n 'username:password' | base64 # For GCloud echo -n "_json_key:$(cat gcloudauth.json)" | base64 -w0- Docker Hub: Use
Enable lifecycle hooks in Watchtower
mainLifecycle hooks allow you to execute shell commands inside a container before or after an update. This feature is disabled by default. To enable it, use one of the following methods:
- CLI Flag: Add
--enable-lifecycle-hooksto your Watchtower command. - Environment Variable: Set
WATCHTOWER_LIFECYCLE_HOOKStotrue.
Important Requirements & Constraints:
- Shell Availability: Commands are executed using
sh, so the target container must include theshexecutable. - Container State: If a container is not currently running, lifecycle hooks will not execute, and the update will proceed without them.
- Failure Behavior: If a command fails (returns an exit code other than
0or75), Watchtower will log the error but will not stop the update process.
# Using environment variable WATCHTOWER_LIFECYCLE_HOOKS=true watchtower # Using CLI flag watchtower --enable-lifecycle-hooks- CLI Flag: Add
Configure Watchtower notifications via Shoutrrr
mainWatchtower uses the Shoutrrr library to send notifications. You can specify one or multiple notification services by providing a space-separated list of Shoutrrr service URLs.
Key Configuration Options
--notification-url(envWATCHTOWER_NOTIFICATION_URL): The Shoutrrr service URL(s). This can also be a path to a file containing the URL(s).--notification-template(envWATCHTOWER_NOTIFICATION_TEMPLATE): A Go text/template used to format the message.--notification-report(envWATCHTOWER_NOTIFICATION_REPORT): When enabled, Watchtower uses the session report data instead of individual log entries for the template.
Important: Multiple Notifications Workaround
Due to a bug in Viper, you cannot use comma-separated values in environment variables for multiple notifications. Instead, use spaces to separate URLs and wrap the value in quotes:
WATCHTOWER_NOTIFICATION_URL="slack://token1 slack://token2"Note: If using
docker-compose, do not use double quotes inside your.ymlfile for these values to avoid startup errors.docker run -d \ --name watchtower \ -v /var/run/docker.sock:/var/run/docker.sock \ -e WATCHTOWER_NOTIFICATION_URL="discord://token@channel slack://watchtower@token-a/token-b/token-c" \ -e WATCHTOWER_NOTIFICATION_TEMPLATE="{{range .}}{{.Time.Format \"2006-01-02 15:04:05\"}} ({{.Level}}): {{.Message}}{{println}}{{end}}" \ containrrr/watchtowerConfigure custom stop signals for containers
mainBy default, Watchtower sends a
SIGTERMsignal to stop a container when an update is detected. If your application requires a different signal to shut down gracefully, you can specify it using the labelcom.centurylinklabs.watchtower.stop-signal.You can apply this label either during the image build process in your
Dockerfileor when starting the container via thedocker runcommand.### Using a Dockerfile ```docker LABEL com.centurylinklabs.watchtower.stop-signal="SIGHUP"Using docker run
docker run -d --label=com.centurylinklabs.watchtower.stop-signal=SIGHUP someimage