Menu
Index

Contact
LinkedIn
GitHub
Atom Feed
Comments Atom Feed



Tweet

Similar Articles

31/10/2009 14:46
SMART stats on Cacti (via SNMP)
08/01/2015 22:58
Nginx on Cacti via SNMP
22/11/2009 16:49
Postfix stats on Cacti (via SNMP)
08/01/2015 22:59
php-fpm on Cacti via SNMP
14/11/2009 13:46
Apache stats on Cacti (via SNMP)
13/02/2015 22:51
opendkim on Cacti via SNMP

Recent Articles

23/04/2017 14:21
Raspberry Pi SD Card Test
07/04/2017 10:54
DNS Firewall (blackhole malicious, like Pi-hole) with bind9
28/03/2017 13:07
Kubernetes to learn Part 4
23/03/2017 16:09
Kubernetes to learn Part 3
21/03/2017 15:18
Kubernetes to learn Part 2

Glen Pitt-Pladdy :: Blog

Linux (Debian, Ubuntu) SNMP basics

I have recently taken to Cacti, the web based graphing system (and more) for monitoring networks, infrastructure, servers, and just about anything you can turn into data it can digest. While there are other popular tools (eg. Nagios), I prefer Cacti as it is well designed, tidy and easy to adapt to my needs. There are various plugins available for Cacti that allow for many scenarios (including automated monitoring and raising alerts when parameters go out of a predefined range), and it is easily extended to take in data from a multitude of sources including easily writing scripts to bring in data from just about anywhere.

Currently I am involved in overseeing deployment of a load of new mission critical infrastructure, and knowing how it is performing and being able to pre-empt any problems as well as autopsy failures we can't anticipate is vital in any environment where business depends on IT to this extent.

SNMP (Simple Network Management Protocol) is the standard for remotely monitoring devices, and the Net-SNMP packages (in Debian & Ubuntu the snmp and snmpd packages) provide an extensible way to script up monitoring just about anything on a box.

SNMP Basics

Without getting into anything too complicated, for simple monitoring all we need to know is that SNMP uses a hierarchy of named (or numbered) objects that can be requested or traversed.

There are 2 main versions of SNMP worth knowing about that are widely used today: v2c and v3. v2c gives basic IP address based security, where v3 brings in cryptography for authenticating the connection and then encrypting the data. Unless you have good reason to do otherwise (eg. your device does not support it), I prefer using v3 and take advantage of the strong authentication and encryption, especially when devices are monitored via an untrusted network.

I am also a believer in providing some basic firewalling (packet filtering) to limit connections to the machine doing the monitoring (SNMP is on port 161). While this is not completely fool-proof, it does reduce the risk of attacks on the SNMP service, especially by "dumb" attackers like worms.

Installing snmpd and the matter of MIBs

For Debian and derivatives the server package mentioned above is snmpd. In most cases you will also want client / admin tools which are in the snmp package.

Now that will get you a basically functioning service, but since MIBs are can't be distributed due to licensing, they can't be part of the "open" packages Debian ships and since I originally wrote this article they have been removed. This means that things that are taken for granted here won't actually work unless you install the snmp-mibs-downloader package as well. This will pull in all the necessary MIBs in place, but to actually use them you will need to look in /etc/snmp/snmp.conf and see the comment there.

By default use of MIBs is disabled so you will need to comment the line as described in the file to enable use of the MIBs.

Collecting data for SNMP

Debian and Ubuntu, keeping with good practice, have created an unprivileged user for running SNMP under. This is very good from a security point of view, it does mean that snmpd can't directly access any data that requires privilege on the system.

There are some options:

Using SUID programs / scripts
What this means is that a program or script may be run by anyone with access determined by the regular file permissions, but the program will be run with with the privilege of the owner, in this case root. Useful, but there is always a risk that someone malicious may discover a way to exploit a flaw and gain privilege on the system by it.
Using a CRON job
CRON can periodically run a script to collect the privileged data and dump it into a file. Scripts run by snmpd can then pick up this data without needing privilege. This means that no risky SUID stuff is needed, but does have the slight disadvantage of data being only as up to date as the last run of the CRON job.

As those who know me could predict, I chose the secure option of running a CRON job to collect the data.

I created a basic shell script in /etc/snmp/local-snmp-cronjob which is written carefully to avoid security problems (all absolute paths, careful escaping and quoting of external data etc.).

I created a custom crontab (/etc/cron.d/local-snmp) with the a single entry to run the CRON job every 5 minutes:

*/5 * * * * root /etc/snmp/local-snmp-cronjob

Many similar scripts on the web store data in /tmp or similar. This is bad! Carelessly writing files as a privileged user to /tmp or any publicly writeable areas invites symlink attacks, and indeed almost all that I looked at before deciding to roll my own where vulnerable to symlink attacks.

How a symlink attack works is that the attacker creates a symlink to some file that they want to attack (generally to damage or overwrite) and name the symlink in /tmp (or other world writeable area) matching a file that they know some program running as a privileged user will try to write. Some of these attacks rely on having to retry if the filename is only partially predictable (eg. includes the PID in the name), or exploit a race condition (eg. checking for the file before opening, rather than using O_EXCL to ensure that existing files aren't overwritten).

To avoid this, I created the directory /var/local/snmp for storing this data which is only writeable by root and so safe to use from basic scripts without careful checks.

Extending SNMP

Net-SNMP provides a simple way of adding custom extensions. This allows scripts and programs to be written which can capture just about any parameters and relay them back via SNMP. Extensions can be added by adding lines to /etc/snmp/snmpd.conf in the format:

extend NAME PATH_TO_SCRIPT OPTIONAL_SCRIPT_ARGS

The script simply dumps a load of values which may be accessed on the SNMP OID of:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."NAME".VALUENUMBER

That means that if our script with the extension name of "foo" returns the numbers:

4.3
68.9
42

Then requesting the following SNMP OIDs will give:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".1 gives 4.3
NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".2 gives 68.9
NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".3 gives 42

SNMP v3 Configuration

Unless you have good reason otherwise, I prefer using SNMP v3 to take advantage of the added security. This requires setting up two pass phrases for authentication and encryption of the data. I generally use "pwgen --secure 16" to create random 16 character pass phrases.

On the system being monitored the authentication is then set up (read only access) with the command:

$ net-snmp-config --create-snmpv3-user -ro -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS USERNAME

AUTHPROTO is the authentication hashing protocol / algorithm and may be either MD5 or SHA. PRIVPROTO is the privacy encryption protocol / algorithm and may be either DES or AES. AUTHPASS AND PRIVPASS are the authentication and privacy passwords, and USERNAME is the SNMP USERNAME for the user you are adding. This should be sufficient with the default config to monitor a host.

To test that it is working, on the host that is doing the monitoring you can connect try:

$ snmpwalk -v 3 -u USERNAME -l authPriv -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS MONITOREDHOSTNAME

If you are monitoring remotely (ie. using a MONITOREDHOSTNAME other than "localhost") then you may also need to remove the "127.0.0.1" from /etc/default/snmpd and restart snmpd to ensure that snmpd is listening on all addresses and not just on the local loopback.

You can add the OID / part of the OID after that to walk a particular section:

$ snmpwalk -v 3 -u USERNAME -l authPriv -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS MONITOREDHOSTNAME NET-SNMP-EXTEND-MIB::nsExtendOutLine

SNMP v2c Configuration

v2c does not have any strong authentication or encryption of the data and as such I tend to avoid it. The basic concept is that a "community" of trusted IP addresses is created which are allowed to connect.

By default snmpd should respond to the community "public" with a subset of read-only parameters (paranoid mode). If you need to to give further access via v2c then this may be done with a combination of the "com2sec", "group", "view" and "access" parameters in /etc/snmp/snmpd.conf. The example config and manual pages have more detail on this if you need to do it.  For example, to give read-only access from the local machine, add this line to the top of the "com2sec" rules:

com2sec readonly 127.0.0.1 public

To test v2c access is working, like above you can use the command:

$ snmpwalk -v 2c -c public MONITOREDHOSTNAME

Again, if you are monitoring remotely (ie. using a MONITOREDHOSTNAME other than "localhost") then you may also need to remove the "127.0.0.1" from /etc/default/snmpd and restart snmpd to ensure that snmpd is listening on all addresses and not just on the local loopback.

Next

I am going to publish the SNMP config and Cacti templates I have created for monitoring various aspects of systems. I will list them here as I publish them:

Comments:

J.Korvin Image  22/08/2011 07:25 :: J.Korvin

I can not understand what should be in a folder /var/local/snmp/mail

Glen Pitt-Pladdy Image  22/08/2011 07:39 :: Glen Pitt-Pladdy

The directory (folder) is /var/local/snmp which is used for storing data which is collected by the cron job. It works like this:

Cron job collects data only available to root -> Creates data file in /var/local/snmp. Script used by snmpd to pick up the data does not have root privilege so can't access the data directly -> picks up data from cron job in /var/local/snmp.

The file /var/local/snmp/mail is used to store stats picked up from the mail system logs (see other article) and from there the various extend scripts pick the data up for snmpd.

I. Bar-Haim Image  24/08/2011 11:59 :: I. Bar-Haim

Am I supposed to get specific data results then running one of the "dovecot-stats-*" scripts. I'm asking because the outputs are "0" during my direct script run.
Maybe you can publish an example of /var/local/snmp/mail typical content.

BTW: Internal data source names of the cacti template are different from names in "extend". I suppose it was just an example of snmpd config.

Glen Pitt-Pladdy Image  24/08/2011 20:32 :: Glen Pitt-Pladdy

The stats scripts for dovecot, postfix etc. just pick up the data in /var/local/snmp/mail so whatever is in there will be reflected by stats scripts. If the cron job that parses the log files is working then /var/local/snmp/mail should contain a load of data.

The scripts are simply pass the field in /var/local/snmp/mail to snmpd to deliver. The field names are not important - the stats scripts just output the numbers in /var/local/snmp/mail:

/etc/snmp$ ./dovecot-stats-auth
63
9431
2
0
175

I. Bar-Haim Image  29/08/2011 07:41 :: I. Bar-Haim

I understand that. Seems like the cron job isn't working as expected. See the exmple of output from my "/var/local/snmp/mail":

lastrun=1314599701
lastline=20019
lastposition=1803685
lastinode=786887

It doesn't look like correct data.

Glen Pitt-Pladdy Image  29/08/2011 08:17 :: Glen Pitt-Pladdy

Your are correct - that is only the logfile info. What is the command line you are running in the cron job to generate this?

I. Bar-Haim Image  29/08/2011 15:21 :: I. Bar-Haim

# Cacti dovecot stats file update - every 5 minutes
*/5 * * * * root /etc/snmp/uloganalyser /var/log/dovecot/dovecot.log.1 /var/log/dovecot/dovecot.log /var/local/snmp/mail dovecot

Location of dovecot.pm plugin:
/etc/snmp/uloganalyser-plugin/dovecot.pm (-rwxr-xr-x)

Glen Pitt-Pladdy Image  29/08/2011 16:10 :: Glen Pitt-Pladdy

That all looks good. I would also expect /var/local/snmp/mail to have a timestamp on it matching the cron job. It's not necessary to make the plugin executable (it's included not executed).

The only remaining thing other than trying to debug what is happening is to check that there is in fact useful things going into the dovecot.log file like:

Aug 28 08:48:50 mailserver dovecot: imap-login: Login: user=<luser>, method=PLAIN, rip=192.168.33.27, lip=192.168.22.50

Also check that in /var/local/snmp/mail dovecot all the last---- values change on each run and things like lastposition should match the filesize in bytes (or at least up to the last line read).

Also worth running the command on the command line manually and see if it gives any errors / warnings.

I. Bar-Haim Image  30/08/2011 08:01 :: I. Bar-Haim

I think it is log parsing issue of the plugin.
What is the dovecot version you've written the plugin for?
Here is my dovecot (v2.0.12) log example:

2011-08-30 09:42:30 imap-login: Info: Login: user=<username>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=30069, TLS
2011-08-30 09:42:30 imap(username): Info: Disconnected: Logged out bytes=91/864
2011-08-30 09:42:33 pop3-login: Info: Login: user=<username>, method=PLAIN, rip=xx.xx.xx.x, lip=xx.xx.xx.x, mpid=30071, TLS
2011-08-30 09:42:33 imap-login: Info: Login: user=<username>, method=PLAIN, rip=xx.xx.xx.x, lip=xx.xx.xx.x, mpid=30073, TLS
2011-08-30 09:42:42 imap-login: Info: Login: user=<username>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=30088, TLS

The file /var/local/snmp/mail values are changing as expected every 5 minutes. No errors or warnings seen during manual run.

Glen Pitt-Pladdy Image  30/08/2011 08:13 :: Glen Pitt-Pladdy

I think the big difference here is that my scripts are meant for logging via syslog (normal way of doing things in Debian/Ubuntu) and in the standard syslog format date/host/source format at the beginning of the line.

You could just change the 3rd line of the analyse sub to match and strip this instead:

if ( $line !~ s/^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+// ) { return; }

I may consider adding that in if more people are logging in different ways.

I. Bar-Haim Image  30/08/2011 08:24 :: I. Bar-Haim

Thanks, it worked.
Systems with more users and traffic are usually have separated log directories to allow better log management.

Glen Pitt-Pladdy Image  30/08/2011 08:32 :: Glen Pitt-Pladdy

Great to hear it.

Logging via syslog allows for separation of log files as well as shipping logs remotely (say to a central logging / analysis node with a cluster).

I. Bar-Haim Image  30/08/2011 08:57 :: I. Bar-Haim

Although it seems to be working, part of the information is not parsed from the log file.
Take a look:

lastrun=1314690601
lastline=44958
lastposition=3942406
dovecot:imap:login=1152
dovecot:imap:login:other=1152
dovecot:pop3:login=233
dovecot:pop3:login:other=233
lastinode=786880

Some lines are not parsed and probably this is the reason they are missing.
I see a lot of similar to those (last condition of the plugin):

/etc/snmp/uloganalyser-plugin/dovecot.pm 20110626:155 /var/log/dovecot/dovecot.log:45368 unknown dovecot: 2011-08-30 10:53:20 imap(username): Info: Disconnected: Logged out bytes=921/82880

Christian Image  29/11/2016 12:39 :: Christian

I get error "./apache-stats.sh: line 47: *1024: syntax error: operand expected (error token is "*1024")" when trying run apache.stats.sh, please your advice for this.Thanks




Are you human? (reduces spam)
Note: Identity details will be stored in a cookie. Posts may not appear immediately