Configuring rsyslog to rotate log files from log messages streamed to it from a Systemd service

In general, I have moved to writing all of my applications to write their log output to STDOUT. This makes running them on the command line, in an IDE, on a bare metal box, VM, or in a container completely decoupled from how you store and view the logs. No more having multiple logging configs for each flavor of deployment.

In this particular case, I am running an application in a container (but it isn’t necessary that it is in a container) controlled by systemd and using rsyslog to forward all of the log messages to a specific output file. A requirement of writing log files to a local disk is that you must be able to rotate and truncate them by size so that you don’t fill up your disk; in either normal operation or some error condition that ends up inadvertently generating a large amount of log messages in a short period of time.

For the following example, we will us the service identifier my_program_identifier. You will update this to define something relevant to your deployment.

To configure your service in this manner you first need to add the appropriate options to the [Service] section of your unit file.

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=my_program_identifier

Then you define an rsyslog.d config file as follows for my_program.conf

$outchannel my_program_log_rotation,/var/log/my_program/my_program.log, 1073741824, /etc/my_program/log-rotate.sh

if $programname == "my_program_identifier" then :omfile:$my_program_log_rotation
& stop

In the rsyslog conf file we define an Output Channel. The TLDR; is that an output channel enables you to define the file name to which you want to write, the max size (in bytes) and a command (or path to a script or program) to run when the file reaches the limit.

In the previous example, we declare an output channel with the $outchannel directive. We then give it the identifier my_program_log_rotation. Then define the path of the log file, the max size, and a shell script that will run to rotate the file for us.

The next line defines how to act upon each of the log messages with the "my_program_identifier" that we defined in the unit file.

Following is a working sample of the log-rotate.sh script.

#!/bin/bash
  
LOG_DIR=/var/log/my_program
FILE_NAME=my_program.log
MAX_NUM_FILES=10

for i in `ls -1 $LOG_DIR/${FILE_NAME}.* | sort --field-separator=. -k3 -nr`
do
  # Grab the number (last token) for all of the numbered files
  log_num=$(echo "$i" | awk -F\. '{print $NF}')

  # If it is equal to or greater than our max number of files
  # just delete it.
  if [ "$log_num" -ge $MAX_NUM_FILES ]
  then
    rm $i
    continue
  fi

  target_num=$((log_num + 1))
  target_file_name="$LOG_DIR/${FILE_NAME}.${target_num}"
  mv -f $i $target_file_name
done

mv -f $LOG_DIR/$FILE_NAME $LOG_DIR/${FILE_NAME}.1

Deploy your updated unit file, your rsyslog.d conf file, and the shell script and you should have it up and running.