Docker logging to the local OS that works, with compose and rsyslog
My objective is to get logging output from my container applications consolidated under /var/log, with logfile names matching their origin container. Logfile cohabitation will allow me to incorporate them in the housekeeping rotation ritual of the rest of my system,
My objective is to get logging output from my container applications consolidated under /var/log, with logfile names matching their origin container. Logfile cohabitation will allow me to incorporate them in the housekeeping rotation ritual of the rest of my system, and help with speedy fault diagnosis.
The environment: Traefik v1, Docker, Docker-Compose, Nginx, Ubuntu 18.04, rsyslogd, logrotate. This is a basic setup presenting the "Welcome to nginx!" page.
Some Docker logging basics
With some standard containers running, some simple logging can be seen using the docker logs command at the prompt, specifying a container name. Here we just see a bunch of simple HTTP 200 responses for the Nginx basic "Welcome to nginx!".
ubuntu@ip-10-10-10-10:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0bf87249ca62 nginx:alpine "nginx -g 'daemon of…" 2 minutes ago Up 2 minutes 80/tcp, 0.0.0.0:8081->8081/tcp traefik_nginx_1
54ac5bebba0c traefik:alpine "/entrypoint.sh trae…" 2 weeks ago Up About an hour 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp traefik_traefik_1
ubuntu@ip-10-10-10-10:~$
ubuntu@ip-10-10-10-10:~$ docker logs traefik_nginx_1
172.0.0.2 - - [07/Sep/2019:07:40:22 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "11.12.13.14"
172.0.0.2 - - [07/Sep/2019:07:40:25 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "11.12.13.14"
172.0.0.2 - - [07/Sep/2019:07:40:25 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "11.12.13.14"
172.0.0.2 - - [07/Sep/2019:07:40:26 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "11.12.13.14"
...I am using a very basic Docker config with docker-compose serving up Traefik and Nginx containers, with the default Welcome screen.
Configuring a Syslog catcher for custom log files
For maximum flexibility, I would like to be able to specify a tag in the compose definition that determines the eventual log file name, and I would like to group all of the container based application logs under a new directory, /var/log/containers. A subdirectory is optional.
In /etc/rsyslog.d I create a new template file, the name is arbitrary, here 40-docker.conf, but check the below text about duplicates when naming. With your favourite text editor add the contents:
# Create a template for the target log file
$template CUSTOM_LOGS,"/var/log/containers/%programname%.log"
if $programname startswith 'docker-' then ?CUSTOM_LOGS
& ~I will prefix any container logs with the string "docker-" for filtering and grouping purposes. Run the following, if you are going to also use a logging subdirectory strategy:
$ sudo mkdir /var/log/containers
$ sudo chown syslog:adm /var/log/containers
$ sudo chmod 775 /var/log/containersRestart rsyslogd with sudo systemctl restart rsyslog (Ubuntu 18.04).
About duplicate log file entries
Syslog will read the files in /etc/rsyslog.d in order. Our sample .conf file has a tilde at the end, which indicates a stop to further processing. Therefore, if you create a .conf file such as "40-docker.conf" that is processed before the 50-default.conf file, it will be logged to the appropriate custom container log, then processing for that entry will stop.
If you create a .conf file such as "60-docker.conf" it will be processed after the 50-default.conf, so the entry will first be logged in the main default syslog, then in the custom docker file. You will end up with the same entry in both places.
Setting up Docker Compose for the logging
I want to use the syslog format, so my logging entry in the compose file for Nginx looks like this.
logging:
driver: syslog
options:
tag: docker-mynginx01There is more info on the syslog driver syntax and options here: https://docs.docker.com/config/containers/logging/syslog/
After adding, and doing a docker-compose up -d, then an Nginx page reload, I am rewarded with a custom log file. I can quickly see from the file name which container it's from, which will be great for debugging.
ubuntu@ip-10-10-10-10:~$ ll /var/log/containers/
total 12
drwxrwxr-x 2 syslog adm 4096 Sep 7 20:26 ./
drwxrwxr-x 11 root syslog 4096 Sep 7 20:07 ../
-rw-r----- 1 syslog adm 258 Sep 7 20:26 docker-mynginx01.log
ubuntu@ip-10-10-10-10:~$
ubuntu@ip-10-10-10-10:~$ cat /var/log/containers/docker-mynginx01.log
Sep 7 20:26:16 ip-10-10-10-11 docker-mynginx01[868]: 172.0.0.1 - - [07/Sep/2019:08:26:16 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "11.12.13.14"
ubuntu@ip-10-10-10-10:~$Set up log rotation for the docker logs
This is straight up Linux stuff. The example here is just a starter rotation config applying to all files in the containers log directory. There are a ton of options, see the man page for logrotate.
With your favourite text editor, create a file in /etc/logrotate.d called, say, docker. Add these contents, change as you see fit. Wildcards should be used with caution.
/var/log/containers/*.log {
daily
rotate 10
missingok
compress
copytruncate
}I've checked, and logrotate works with files owned by syslog or root, so probably the logfile owner doesn't matter. I did have some problems with logs failing to write after rotation, the copytruncate parameter fixed it.
Cron will parse the rotation files daily.
An optional flag to shift Ghost logs from the container
I found that Ghost was logging inside the container. I know, EUW, right? Unbeknownst to me, Ghost had created GBs of logs under [docker mount]/logs , without rotation.
Adding a forced STDOUT to the docker compose file fixed it, e.g.
environment:
logging__transports: '["stdout"]'This is a Ghost-specific flag, dig into your container provider docs to see if they have a similar flag to log to STDOUT and tame your naughty comtainers.
Apply custom logging to all the container things
Once proven, the "logging" option can be - cautiously - applied to all of your docker-compose services. Verify the log rotation after a day or so.
Having the logs on hand, all in one place, will allow you to really fine tune your configuration. Often there are many small errors that won't stop an application from working, but could potentially in future, and might indicate server errors or security issues.
To illustrate the point, when I enabled logging for my Traefik container, I discovered an "Error renewing certificate from LE" error that had been going on the whole life of the server. It was functioning, but constantly faulting. It took less than a minute to fix.
If you are not looking at Nginx or Trafik logging specifics, the next couple of chapters can be skipped, feel free to jump forward to "It Works!".
A note about Nginx application logging
This chapter is about more advanced stacks, and not about the demo build example.
If you have followed some of my previous posts, or it just is, you might have local Nginx configuration, not using syslog. In that situation, you might have a volume mapping in your docker compose a bit like: /data/logs/nginx-wp1:/var/log/nginx in combination with an Nginx config file containing something like:
access_log /var/log/nginx/wp1-access.log;
error_log /var/log/nginx/wp1-error.log;In my case I would like to keep the separation between access and error logs, but still benefit from both log rotation and cohabitation - having the logs coexist with the rest of my containers' logs. So rather than stdout and rsyslog, I am going to map the log volumes directly.
This is optional, the default stdout logging also works fine as per the example compose file at the end - feel free to skip this section.
Map the container log directory to the Nginx container
Volume map before:
volumes:
- /data/nginx-wp1:/etc/nginx/conf.d
- /data/nginx-wp1/nginx.conf.new:/etc/nginx/nginx.conf
- /data/logs/nginx-wp1:/var/log/nginx
- /data/wp1:/var/www/htmlVolume mapping after:
volumes:
- /data/nginx-wp1:/etc/nginx/conf.d
- /data/nginx-wp1/nginx.conf.new:/etc/nginx/nginx.conf
- /var/log/containers:/var/log/nginx
- /data/wp1:/var/www/htmlTweak the log file names in the Nginx default.conf
Now that the log files will be cohabiting with the other container logs, I will modify the file names a little.
Log file names before:
access_log /var/log/nginx/wp1-access.log;
error_log /var/log/nginx/wp1-error.log;Log file names after:
access_log /var/log/nginx/docker-nginx-wp1-access.log;
error_log /var/log/nginx/docker-nginx-wp1-error.log;After the mods, I regenerate with compose, and note that the server has gone into a bit of an out of memory death spiral. At this point I remember it's just a 1GB server running double Bloatpress, and there is just a few hundred MB memory headroom. I reboot, then go for it again, successfully this time.
I have another post on reducing memory use.
Nginx Log File permissions
The Nginx logs have been created as root, so rather than wait and see if log rotate breaks, I run a sudo chown syslog:adm on them.
A note about Traefik logging
In order for the traefik logging to work in this way, it needs to be logging to stdout. I found that my config initially was not. Check your traefik.toml file (or equivalent) and verify that the [traefikLog] parameter is unset / blank.
If the parameter is not there at all, add it with nothing in it, then restart the container. This channel is pretty quiet if your config is clean, so if you don't see a traefik log file generated after setting it up, it might not be a problem.
If you would like ACCESS logs, do the same but add the [accessLog] parameter. No further values will enable with default settings. These logs will of course be super noisy, so watch out for disk fill risk. More info here: https://docs.traefik.io/configuration/logs/
Having both seems to munge both into one file - as you would expect with stdout logging.
It works!
I check after a day, and the log files, and rotation looks good. Your reward will be files cycling and compressing daily.
$ ll /var/log/containers
total 20
drwxrwxr-x 2 syslog adm 4096 Sep 10 06:25 ./
drwxrwxr-x 11 root syslog 4096 Sep 10 06:25 ../
-rw-r----- 1 syslog adm 0 Sep 10 06:25 docker-mynginx01.log
-rw-r----- 1 syslog adm 559 Sep 9 18:38 docker-mynginx01.log.1.gz
-rw-r----- 1 syslog adm 0 Sep 10 06:25 docker-mytraefik01.log
-rw-r----- 1 syslog adm 424 Sep 8 16:35 docker-mytraefik01.log.1.gz
...I have to confess that before I established this, it felt like I was flying blind a bit with containerised apps. Now I have control! It's also given me a huge amount of info to browse through and look for opportunities to enhance the configuration.
Example full docker-compose file, with logging
- This is a simplified Nginx config with no volume mapping or custom config files
- I have chosen the "maroilles-alpine" traefik image to fix this at V1. You could simply have
image: traefik:alpineor even justimage: traefik, but at some point that will become V2 which is not the config I have based this on.
version: "3"
networks:
web:
external: true
services:
traefik:
image: traefik:maroilles-alpine
restart: always
logging:
driver: syslog
options:
tag: docker-mytraefik01
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /data/traefik/traefik.toml:/traefik.toml
- /data/traefik/acme.json:/acme.json
labels:
- traefik.frontend.rule=Host:monitor.yourhost.com
- traefik.port=8080
networks:
- web
nginx:
image: nginx:alpine
restart: unless-stopped
logging:
driver: syslog
options:
tag: docker-mynginx01
ports: # HOST:CONTAINER
- '8081:8081'
labels:
- traefik.backend=nginx
- traefik.frontend.rule=Host:nginx.yourhost.com
- traefik.docker.network=web
networks:
- webBarely relevant log pile photo courtesy of Pär Pärsson on Unsplash
Retrospective blog post to use BlueSky for comments: techroads.org/docker-loggi... #Docker #rsyslog
— TechRoads blog (@techroads.org) Feb 22, 2024 at 11:37 am