Basic Syslog configuration

Posted on: 2021-01-15

In my years of administration in IT, one area that seems to be underutilized is log monitoring. While most enterprises have some sort of centralized logging solution, whether it's an ELK stack or commercial product like Datadog, it isn't uncommon to have plenty of hosts deployed with their system logs left on the local disk, with no monitoring, no alert and no one looking at them. Even if they are sent to some central server, no one looks at them and there are rarely any filters set up. To me, logs are meant to warn of potential issues. They can be useful to help when reacting to an incident, but ideally you should not be collecting them just to index them somewhere out of sight, you should have filters with alerts when an error or critical situation occurs.

Linux already comes with a very capable logging system called Syslog. Modern distributions use the SystemD version called JournalD, but for this we'll want the Syslog version because it allows more flexibility including forwarding logs to a central server, and adding custom log files to it, in our case software updates. Both JournalD and Syslog can live side by side on the same hosts. You can install it with the apt install rsyslog command.

The goal of this exercise is:

Setup a custom Syslog template.
Set one host as the central logging server.
Set all other hosts as forwarders to the central server.
Include software updates in Syslog.
Setup a basic email alert for any warning, error or critical log entries.

Setting up the server

After you get Syslog installed on all your Linux hosts, pick a central server and configure the /etc/rsyslog.conf file as such:

# Base modules
module(load="imuxsock")
module(load="imklog")
module(load="imfile")

# Server
module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")

# Set default options
template(name="CustomSyslogFormat" type="string" string="%timereported:1:19:date-rfc3339% %hostname% %programname% %syslogseverity-text% - %msg%\n")
global(workDirectory="/var/spool/rsyslog")

# Include apt history
input(type="imfile" File="/var/log/apt/history.log" Tag="apt" Severity="warning" Facility="local7" Ruleset="aptRS")
ruleset(name="aptRS") {
        if ($msg contains "Install:" or $msg contains "Remove:" or $msg contains "Upgrade:") then {
                *.* /var/log/syslog;CustomSyslogFormat
        }
}

# Save all logs locally
*.* /var/log/syslog;CustomSyslogFormat

Once saved, you can restart the Syslog service with the systemctl restart rsyslog command. Basically, in this file we're setting up a few basic modules, enabling the incoming UDP and TCP ports, setting up a template for how the logs will look like, adding the APT history file (filtering with just the install, upgrade and remove entries) and finally we're saving all logs to a single file.

Setting up other nodes

On other nodes, you can configure the /etc/rsyslog.conf file the following way:

# Base modules
module(load="imuxsock")
module(load="imklog")
module(load="imfile")

# Set default options
template(name="CustomSyslogFormat" type="string" string="%timereported:1:19:date-rfc3339% %hostname% %programname% %syslogseverity-text% - %msg%\n")
global(workDirectory="/var/spool/rsyslog")

# Include apt history
input(type="imfile" File="/var/log/apt/history.log" Tag="apt" Severity="warning" Facility="local7" Ruleset="aptRS")
ruleset(name="aptRS") {
        if ($msg contains "Install:" or $msg contains "Remove:" or $msg contains "Upgrade:") then {
                *.* /var/log/syslog;CustomSyslogFormat
                *.* @logs.dendory.net:514
        }
}

# Save all logs locally
*.* /var/log/syslog;CustomSyslogFormat

# Forward to central log server
*.* @logs.dendory.net:514

Here, you should change logs.dendory.net with the address of your central server. If you go through the config file, it's very similar to the server, except we're forwarding the logs instead of accepting incoming connections.

Once this is all set up, you can look at /var/log/syslog on the central server and you should see all the system logs for all of your Linux hosts:

2025-01-14T15:30:17 firefly systemd info -  Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
2021-01-14T15:30:17 firefly systemd info -  Removed slice user-0.slice - User Slice of UID 0.
2021-01-14T15:30:17 firefly systemd info -  user-0.slice: Consumed 3.495s CPU time.
2021-01-14T15:30:18 apps CRON info -  (CRON) info (No MTA installed, discarding output)
2021-01-14T15:30:18 apps CRON info -  pam_unix(cron:session): session closed for user root
2021-01-14T15:30:19 portainer apt warning -  Remove: lm-sensors:amd64 (1:2.6.0-5.1)
2021-01-14T15:30:21 fauna systemd info -  user-runtime-dir@0.service: Deactivated successfully.
2021-01-14T15:30:21 fauna systemd info -  Stopped user-runtime-dir@0.service - User Runtime Directory /run/user/0.
2021-01-14T15:30:21 fauna systemd info -  Removed slice user-0.slice - User Slice of UID 0.
2021-01-14T15:30:21 fauna systemd info -  user-0.slice: Consumed 3.503s CPU time.
2021-01-14T15:30:24 adusa CRON info -  (CRON) info (No MTA installed, discarding output)
2021-01-14T15:30:24 adusa CRON info -  pam_unix(cron:session): session closed for user root

As you can see, each log entry is saved in a single line and includes the timeststamp, the hostname, the process name, the severity level, and then the actual log message. Software updates, such as the removal of the lm-sensors package, are also logged with the warning severity.

Notifications

At this point, you can easily forward that central log into whatever centralized logging solution you have. You could also setup some simple notifications using a shell script, so that you receive a daily email with any warning, error or critical entry:

#!/bin/bash

grep -E '^\S+\s+\S+\s+\S+\s+(warning|err|critical)\s+' /var/log/syslog > /tmp/errors.log

if [ -s "/tmp/errors.log" ]; then
	mail -s "Errors log entries found" "youremail@example.com" < "/tmp/errors.log"
fi

rm -f "/tmp/errors.log"

This script uses grep to parse the severities you're interested in, and then uses the mail command to send the log by email. Make sure you enter your email address, and then use a Crontab entry to run it daily.

While this solution may not be a replacement for enterprise logging solutions, especially since you don't get the fancy dashboards, graphs, or plugins to integrate with other types of logs that aren't compatible with Syslog, this solution can be implemented in minutes and is far better than having your system logs not being monitored by anybody.