I'm a great believer in monitoring things closely and use logcheck on servers to report any abnormal behaviour. This can be anything from a disk starting to fail to malicious attacks, but it's good to know about it.

No such tool seems to exist on OpenWrt, so I set about writing my own. The key thing here is that it's very lightweight, and best if it's a shell script which runs with busybox.  This is my "mini_logcheck" script and how to use it.

Concept

The idea behind logcheck is that is provides a set of rules (in regex) which are then used to parse new log lines periodically, and report back by mail. I for OpenWrt I wanted a much simpler log parser which only has one set of rules - what to ignore.

I also don't want binary dependencies like logtail.

How it works

For OpenWrt I use tail to get new lines which does have the disadvantages that it is line based which is inefficient (reads the whole file each time) and it can't handle partial lines if it happens to be reading while the logfile is being written. This is however minor risk compared to busybox syslogd's tenancy for truncating lines so I'm not going to worry about it.

We store a position log in /var/log/messages.pos which has both the next line number to read and the inode of the log file we last read. This is important as if the files get rotated we can detect it easily by looking at the inode. Again, there is an unavoidable race condition with log rotation since we can't be sure if the logfile gets rotated during the read.

Rules are stored in /etc/logcheck.ignore.d and lines beginning '#' and empty lines are ignored and then the pattern is applied to ignore all the matching patterns in the logs.

What remains (if anything) is emailed on.

Installation & Configuration

Download mini_logcheck for OpenWrt and ensure you also have the following packages installed:

• cron - needed to run mini_logcheck
• mini_sendmail - needed to send emails

Put mini_logcheck somewhere suitable (eg. /usr/sbin though /etc would arguably be OK considering it contains email configuration). You will need to edit the file to set the email configuration appropriately for your needs.

Create the directory /etc/logcheck.ignore.d where your exclusion rules would be stored. You may like to put an initial rule in a file here for crond:

crond\[[0-9]+\]: USER root pid [0-9]+ cmd /usr/sbin/mini_logcheck$

This will be logged each time the cron runs the script so if you don't have this you will get an email every time.

Then add a cron entry (create a a new file if needed) in /etc/crontabs/root to run the script:

# m h dom mon dow command
*/30 * * * * /usr/sbin/mini_logcheck

How often you run the check is up to you - more often will catch problems earlier but risks filling your mailbox with messages if you don't have your rules right.

Restart cron to read the new crontab and start things off:

# /etc/init.d/cron restart

Then each time the script runs it should email you any new log lines which are not excluded by the patterns.

You can create patterns in files in /etc/logcheck.ignore.d and organise patterns named files for different types of messages as suits you. The patterns are all enhanced regex (grep -E) which makes matching some things easier. They are however case sensitive so keep that in mind when writing exclusion patterns.

It's wise to make your patterns as specific as possible as if they match something in a different line then that line will also be excluded and may be something you really should know about.

Making syslogd log to a file

The default busybox syslogd in OpenWrt logs to a circular memory buffer - for this to work we need it to go to a file, but fortunately that's easy.

The file /etc/config/system contains the configuration for syslogd. Under the "system" section add the following option:

        option log_type file

By default syslogd will rotate log files larger than 16KB which is rather small so you may want to increase this, but keep in mind you can have double whatever figure you choose used by log files and on some OpenWrt devices there isn't a log of memory available to store log files. For example, for 2MB log files add the option:

        option log_size 2048

Then you need to restart syslogd, but be careful here. The safest method is to reboot the router, but you may be OK running:

# /etc/init.d/boot restart

Be careful however - this may also mess up your network config and much more.

As mentioned, busybox syslogd (and/or klogd) have problems with long log lines and end up truncating them and generally mangling lines. If this is a problem for you then you may want to consider switching to syslog-ng rather than the busybox syslogd & klogd.