Scheduling a Docker Compose image auto-update in linux

My objective is to have a small script run daily which pulls any new Docker image versions and runs a docker-compose update, capturing the output to a log. The environment is Ubuntu 18.04.

When I connected in to the systems to remediate, often all that was needed was a docker-compose up -d and everything would start working again.

Subsequently, I found that a better way to update is simply a docker-compose pull --no-parallel followed by a docker-compose up -d, so I am setting out to automate it.

Setting image tags in the compose file

For this to be effective, the docker-compose.yaml file must have considered tags for the images. For example, my image entry for the Ghost container has image: ghost:2-alpine because I do not want uncontrolled updates beyond version 2 without testing first. Using this tag should always pull the latest version 2 which is the risk level I am happy with.

If I was to have image: ghost or image: ghost:latest I would be auto updated to version 3 without knowing it. This is your judgement - set your tags to something appropriate.

If it's Wordpress

For Wordpress, check the image you have in your compose is the latest, and is the same "theme" (fpm, Apache, Alpine etc).

I think the best approach is to update the image, confirm the WP version that's in it, then as the version does not seem to auto update inside the app, afterwards update on the WP panel. At least for me, this sequence didn't seem to break things.

Additionally I am adding a "restart" to the compose commands as I have had issues in the past with wordpress image updates.

Test before automating

As I recently found out, it's a good idea to back up your server (this was a Lightsail snapshot in my case) before starting, and to run a pull manually at the command line before committing a cron job.

Manually run the pull and, (if it's clean) the update at the command line.

Warning: If you have never done this or it's been a long time, it can smash a small server running a pull on everything, when all have a pending image. When doing this recently I crashed a server (with 8 pending images) and had to reboot, and pull each image individually. Once up and running, if doing a daily pull this should not be an issue with one or two at a time. The --no-parallel option should help.

Choose or create a target log file

The system default cron log

In default config, cron will send some logs to the default syslog. Alternatively, you may wish to separate it first by going to /etc/rsyslog.d and editing 50-default.conf with your favourite text editor. In there you will find the following entry. Uncomment it (remove the #) to start cron logging to the specified file.

I don't want duplicates going to the main syslog, so I add a cron.none field to the syslog entry. The two lines end up like so.

*.*;auth,authpriv.none,cron.none        -/var/log/syslog
cron.*                                  /var/log/cron.log

Then restart the log daemon: sudo systemctl restart rsyslog.service. If you leave the cron log name and path as default, it will get picked up by existing logrotate configuration.

A custom log for the update job

Optionally - you can create a custom log file to capture the script output. In my case, I will send to a log in /var/log/containers, a directory I already created in this article. If creating a new directory, you should also update the log rotation settings as described in the article.

At this time you can choose a new log such as /var/log/containers/docker-compose.log otherwise output will just munge to syslog, /var/log/syslog.

Create an update script to run daily, weekly or monthly

I will create a small shell script for this. As you wish it can be placed in /etc/cron.daily, /etc/cron.weekly, or /etc/cron.monthly. I am going to fly by the seat of my pants and run it daily. As far as I can tell, the pull will only pull something if there is an updated image.

First I cd /etc/cron.daily , then create a script I will name docker-compose-update using my favourite text editor. Editing with sudo prefix, I add these contents.

Example 1 - Traefik and legacy Docker Compose

#!/bin/sh

set -e

echo "Commencing docker-compose update `date`" >> /var/log/containers/docker-compose-update.log
# Do a pull then an update
/usr/local/bin/docker-compose -f /data/traefik/docker-compose.yml pull --no-parallel >> /var/log/containers/docker-compose-update.log 2>&1
/usr/local/bin/docker-compose -f /data/traefik/docker-compose.yml up -d >> /var/log/containers/docker-compose-update.log 2>&1
echo "Sleeping 10 seconds." >> /var/log/containers/docker-compose-update.log
sleep 10
/usr/local/bin/docker-compose -f /data/traefik/docker-compose.yml restart >> /var/log/containers/docker-compose-update.log 2>&1
echo "Finishing docker-compose update `date`" >> /var/log/containers/docker-compose-update.log

For a later versions, the /usr/local/bin/docker-compose command gains a space and a new path, becoming /usr/bin/docker compose. This example is from Ubuntu 22.04, and caddy rather than traefik.

Example 2 - Caddy and more recent Docker Compose

#!/bin/sh

set -e

echo "Commencing docker compose update `date`" >> /var/log/containers/docker-compose-update.log
# Do a pull then an update
/usr/bin/docker compose -f /data/caddy/docker-compose.yml pull --no-parallel >> /var/log/containers/docker-compose-update.log 2>&1
/usr/bin/docker compose -f /data/caddy/docker-compose.yml up -d >> /var/log/containers/docker-compose-update.log 2>&1
echo "Sleeping 10 seconds." >> /var/log/containers/docker-compose-update.log
/bin/sleep 10
/usr/bin/docker compose -f /data/caddy/docker-compose.yml restart >> /var/log/containers/docker-compose-update.log 2>&1
echo "Finishing docker compose update `date`" >> /var/log/containers/docker-compose-update.log

Optional "docker prune" to clean up

Old images will build up over time. If you are SURE you don't need to roll back, automating the cleanup of old images can help save an unexpected full disk in the future.

Either add to the script or create a new script with the following.

echo "Commencing docker compose prune `date`" >> /var/log/containers/docker-compose-update.log
/usr/bin/docker image prune -a -f >> /var/log/containers/docker-compose-update.log
/usr/bin/docker container prune -f >> /var/log/containers/docker-compose-update.log
echo "Finishing docker compose prune `date`" >> /var/log/containers/docker-compose-update.log

Check all the paths, and update for your host

• My docker-compose command is at /usr/local/bin/docker-compose or more recently usr/bin/docker
• My chosen log target is /var/log/containers/docker-compose-update.log which will need the directory to exist in order to write to it
• My compose yaml file happens to be at /data/traefik/docker-compose.yml or /data/caddy/docker-compose.yml (though it's nothing to do with Traefik or Caddy, really)

I'm adding a 10 second sleep to allow updated packages to instantiate themselves properly before restarting.

Set cron executable permissions

In the same directory as the script, called say docker-compose-update:

$ sudo chmod 755 docker-compose-update
$ sudo chown root:root docker-compose-update

Test the docker compose update commands

You should be able to 1: run the commands in the script individually at the command line and see it work. That way is safest if you have not updated for a while.

Or 2: run the whole script itself which is a good test but will take a long time if you have a backlog of updates or a lot of services.

Example test run, where your ubuntu user has permission to run docker. If not prefix with sudo:

$ cd /etc/cron.daily
$ ./docker-compose-update

Following a run you should have a log file available to check at say /var/log/containers/docker-compose-update.log.

Make a note to yourself to check after the next cron run - the times of the various scheduled runs can be seen by running cat /etc/crontab.

If your update run has been successful, you should see a log file with something like this. Often this is in the .1 suffixed log due to overnight rotation.

$ cat /var/log/containers/docker-compose-update.log
Commencing docker-compose update Sat Aug 15 09:16:52 NZST 2020
Pulling caddy (caddy:2-alpine)...
2-alpine: Pulling from library/caddy
Digest: sha256:707e960e783141b786754199242c2aa10046712e7b2697aa3bc030492c50de2a
Status: Image is up to date for caddy:2-alpine
Pulling techdbo (nginx:alpine)...
alpine: Pulling from library/nginx
Digest: sha256:a97eb9ecc708c8aa715ccfb5e9338f5456e4b65575daf304f108301f3b497314
Status: Image is up to date for nginx:alpine
Pulling phpt (php:7-fpm-alpine)...
7-fpm-alpine: Pulling from library/php
Digest: sha256:b9628c1dd26165603f75bba116da4bb436b117613895117e34cde8b4ab2f29a3
Status: Image is up to date for php:7-fpm-alpine
Pulling varnish-techroads (varnish:stable)...
stable: Pulling from library/varnish
Digest: sha256:935de9d1818e70b08ed977a6fafcd44e18cc18ed3de41d55ea8b93dcc49ee2d4
Status: Image is up to date for varnish:stable
Pulling ghosttechdbc (ghost:3.22-alpine)...
3.22-alpine: Pulling from library/ghost
Digest: sha256:33b1e9b6be06562b0127beaecb7ffd8c05a2f9202ec24e4bb60f8443afc5480d
Status: Image is up to date for ghost:3.22-alpine
caddy_caddy_1 is up-to-date
caddy_varnish-techroads_1 is up-to-date
caddy_ghosttechdbc_1 is up-to-date
caddy_techdbo_1 is up-to-date
caddy_phpt_1 is up-to-date
Sleeping 10 seconds.
Restarting caddy_techdbo_1           ... done
Restarting caddy_varnish-techroads_1 ... done
Restarting caddy_phpt_1              ... done
Restarting caddy_ghosttechdbc_1      ... done
Restarting caddy_caddy_1             ... done
Finishing docker-compose update Sat Aug 15 09:17:19 NZST 2020

You may have better luck with Watchtower than I did, but this will give you a minimal update option.

Good luck with your road!

Barely relevant diary photo courtesy of Eric Rothermel on Unsplash

Retrospective blog post to use BlueSky for comments: techroads.org/scheduling-a... #DockerCompose

[image or embed]

— TechRoads blog (@techroads.org) Feb 22, 2024 at 11:39 am