Create a custom Linux log file with rotation

Rather than drop my custom logs into the vast cesspool that is syslog, I'd like to create a custom log file and automate the rotation, for an AWS CLI function I am running from cron.

My environment is Ubuntu Linux version 18.04, but many other Linuxes will be the same.

Choosing a log file location

Head to /var/log and have yourself an LS command. You might see output something like this:

/var/log$ ls
alternatives.log  btmp                   kern.log       syslog       syslog.6.gz          wtmp
amazon            cloud-init-output.log  kern.log.1     syslog.1     syslog.7.gz
apt               cloud-init.log         kern.log.2.gz  syslog.2.gz  tallylog
auth.log          dist-upgrade           landscape      syslog.3.gz  ufw.log
auth.log.1        dpkg.log               lastlog        syslog.4.gz  ufw.log.1
auth.log.2.gz     journal                lxd            syslog.5.gz  unattended-upgrades

You will need to decide whether to add to an existing file, create a new log file in the log directory, or make a new directory. In my case I might want more logs related to backups in future, and I just like to be special, so I am going to make a backups directory.

/var/log$ sudo mkdir backups
/var/log$ sudo chmod 750 backups
/var/log$ ll -d backups
drwxr-x--- 2 root root 4096 Apr 18 20:47 backups/

I don't mind if the file owner is root as it will be a root process (root cron) that will be writing to the log.

Testing the command to append output to a log

Really this is completely your thing, but make sure anything you put into a cron job has been manually tested at the command line, as the user who the cron will run at, as this will be root by default. I am in reality kicking off an AWS CLI backup in the form of a sync to an S3 bucket, but for simplicity's sake let's go with the following commands, dumping to the new log file.

echo "Commencing my run. `date`" >> /var/log/backups/mybackupstuff.log
echo "This is my command which will make some output." >> /var/log/backups/mybackupstuff.log 2>&1
echo "Run completed. `date`" >> /var/log/backups/mybackupstuff.log

I run each of these as root, as that is the cron user that will be running the job. I check the log directory and cat the contents of the file, which looks a bit like this.

/var/log/backups# cat mybackupstuff.log
Commencing my run. Thu Apr 18 21:24:42 UTC 2019
This is my command which will make some output.
Run completed. Thu Apr 18 21:26:50 UTC 2019

Nice. Separately I can configure a scheduled job that sends output to my new logfile.

Note - there is also the logger command available which is much better for inline logging. In my case I am choosing a dump of standard output as it's more appropriate for the type of job I have.

Automating the custom log file rotation

This Linux flavour has the logrotate system. cd into /etc and edit the logrotate.conf file with your favourite Linux text editor. Under the comment # system specific logs may be configured here add an entry for your log. The following example uses some basic parameters, have a read of the man page for logrotate to customise.

# system-specific logs may be configured here
/var/log/backups/mybackupstuff.log {
    missingok
    weekly
    rotate 10
    compress
    notifempty
}

You can now dump your text to the log file knowing that housekeeping is in place. As always with automation, check back afterwards to see if everything is working as expected!

Main photo courtesy of Ales Krivec on Unsplash

Retrospective blog post to use BlueSky for comments: techroads.org/create-a-cus...

[image or embed]

— TechRoads blog (@techroads.org) Feb 22, 2024 at 11:26 am