Setup rsyslog Client Forwarder on Raspberry Pi

What is syslog

syslog is a syslog over TCP (new) and syslog over UDP (old) standard for message logging. It allows you to separate the software that generates messages, the system that stores them, and the software that reports and analyzes them. syslog is commonly used in enterprise environments to track logins, errors, and all types of sensitive information. Normally syslog is local, and setup to dump to the /var partition to avoid filling up a small hard drive with log junk. This tutorial will show you how to setup syslog on your raspberry pi to send information to another syslog server (hopefully also with enough disk space and on /var).

Splunk and syslog

Splunk is a free event correlation tool (free for some usage). It allows you index and make sense of huge volumes of messages, alerts, etc. Syslog is one the formats it accepts. My splunk server is running on 192.168.1.14 and I am going to add events from my raspberry pi so it shows up in the splunk dashboard. The first step is enabling syslog forwarding on the Pi. The Pi is already logging to it’s own internal syslog file, but I want to look at all of my computers at once. In order to do this, I have to forward syslog info to the splunk server. I reference splunk often throughout the tutorial because that is my syslog correlator (you are free to use other correlators though).

Syslog on Windows vs Linux

sylog is native on linux and can usually be installed easily. Windows uses event logs, and is not compatible with syslog. However, you can use tools like PRTG to forward event logs over syslog to a remote server for correlations. We aren’t going to cover installing syslog on windows, but the software just mentioned can do that for you. Instead, let’s talk about installing syslog on ubuntu, specifically on a Raspberry Pi.

What Should You Monitor?

This list is not complete, but it is a great start of the various things worth monitoring:

/var/log/messages
Contains most system messages
/var/log/secure
Authentication messages
/var/log/cron
Logs Cron job activities
/var/log/maillog
Mail transactions

You want to log/monitor things related to user issues, hardware, security. Syslog can do all of this.

syslog Marks

In case there are no events to report, you will see — MARK — to let you know the service is still running. Don’t be alarmed.

Possible Splunk Diagram

In the rest of this tutorial, we will use the above diagram as a reference and setup syslog to forward from 192.168.1.12 to 192.168.1.14

Remove rsyslog

In the event you need to uninstall rsyslog, the command is very simple. If you want to remove configuration files for rsyslog, type:

In case you don’t want to touch your old config files, but want to uninstall the binaries for rsyslog:

Install rsyslog Raspberry Pi

Edit rsyslog.conf

The syslog configuration files are located at /etc/rsyslog.conf. You will edit this file on the client for the client portion and on the server for the server portions. You can use any editor you like to edit your syslog.conf file, but I use vim. Other options are: nano, pico, vi. vim has a pretty steep learning curve, but is also one of the most powerful text editors and is usually available on any version of linux you touch. It’s worth learning vim!

You will want to replace the 192.168.1.14 with the IP that matches your syslog server. Your client will send info to that server.

Restart syslog Service

As sudo, restart the syslog process.

Verify rsyslog is Running

You can verify that rsyslog is running by looking at the status:

View Packets of syslog to Verify

You can verify that your client (192.168.1.12 in my example) is sending packets to the server (192.168.1.14 in my example), by using tcpdump.

If you need to install tcpdump on the raspberry pi (or any debian based flavor), type:

After it has been installed, you can view the syslog traffic:

Manually Trigger a Log Event

Linux has a tool called “logger” that lets you dump information into the syslog (useful for testing log functionality).

This sent a syslog event to the remote server instead of to my logs on the Pi because I have already configured the Pi to send to the remote server!

Configure rsyslog Server

Of course, unless your server is listening for syslog, it won’t matter that your client is sending syslog info. If you install splunk, it will have an option to listen for TCP 514, and therefore will be able to collect and index your syslog traffic from all of the clients that it info using the splunk daemons. You can also configure the same /etc/rsyslog.conf file to listen (more info to come in the next article).

Considerations of Syslog

Sensitive Info

Syslog usually has information that could be considered sensitive and UDP syslog is not encrypted. Upgrade to syslog-ng and use tcp with tls or use syslog on a network that you 100% trust. Encrypted syslog messages are recommended.

Centralized Collection

For syslog in general, it’s considered best practice to consolidate all events into a single, hardened host. A syslog server will be a prime target to be attacked. It is advisable to harden it thoroughly, maintain patching schedule and not run other services on the same box.

Use NTP!

Synchronize all servers to the same time zone so you can easily track event times.

Filling Up Partitions

You might have a small partition that could easily fill up with volumes of syslog data. It’s best to use a partition dedicated to log files, like /var that won’t crash your root file system in the event it completely fills up. A backup and export plan using the logrotate program is recommended.

Missing Information

If you configure syslog to go directly to another machine, without logging, you might miss log info if the network or server is down. Configure syslog TCP for buffering at least and try to write to a file then forward file information into a correlation tool, like splunk. Splunk has a listener daemon, but if you are sending UDP traffic, you might miss important events and log info.

Baseline Patterns

Using a tool like splunk, you can detect a baseline and deviations from that baseline. If you aren’t reviewing your logs regularly, they are useless.

Faster Alerts

Many syslog messages are great for audits, but configure your correlation software to alert you on the high priority messages so you can take action.

Prevent DDOS of syslog

syslog can be DOS’d. You can configure your server to only listen to specific hosts and you can also break up logs by facility (daemon, mail, auth, etc) to help sort through noise. Prevent users without reason from using the logger script so they cannot flood the syslog.