Glen Pitt-Pladdy :: Blog

I have recently taken to Cacti, the web based graphing system (and more) for monitoring networks, infrastructure, servers, and just about anything you can turn into data it can digest. While there are other popular tools (eg. Nagios), I prefer Cacti as it is well designed, tidy and easy to adapt to my needs. There are various plugins available for Cacti that allow for many scenarios (including automated monitoring and raising alerts when parameters go out of a predefined range), and it is easily extended to take in data from a multitude of sources including easily writing scripts to bring in data from just about anywhere.

Currently I am involved in overseeing deployment of a load of new mission critical infrastructure, and knowing how it is performing and being able to pre-empt any problems as well as autopsy failures we can't anticipate is vital in any environment where business depends on IT to this extent.

SNMP (Simple Network Management Protocol) is the standard for remotely monitoring devices, and the Net-SNMP packages (in Debian & Ubuntu the snmp and snmpd packages) provide an extensible way to script up monitoring just about anything on a box.

SNMP Basics

Without getting into anything too complicated, for simple monitoring all we need to know is that SNMP uses a hierarchy of named (or numbered) objects that can be requested or traversed.

There are 2 main versions of SNMP worth knowing about that are widely used today: v2c and v3. v2c gives basic IP address based security, where v3 brings in cryptography for authenticating the connection and then encrypting the data. Unless you have good reason to do otherwise (eg. your device does not support it), I prefer using v3 and take advantage of the strong authentication and encryption, especially when devices are monitored via an untrusted network.

I am also a believer in providing some basic firewalling (packet filtering) to limit connections to the machine doing the monitoring (SNMP is on port 161). While this is not completely fool-proof, it does reduce the risk of attacks on the SNMP service, especially by "dumb" attackers like worms.

Installing snmpd and the matter of MIBs

For Debian and derivatives the server package mentioned above is snmpd. In most cases you will also want client / admin tools which are in the snmp package.

Now that will get you a basically functioning service, but since MIBs are can't be distributed due to licensing, they can't be part of the "open" packages Debian ships and since I originally wrote this article they have been removed. This means that things that are taken for granted here won't actually work unless you install the snmp-mibs-downloader package as well. This will pull in all the necessary MIBs in place, but to actually use them you will need to look in /etc/snmp/snmp.conf and see the comment there.

By default use of MIBs is disabled so you will need to comment the line as described in the file to enable use of the MIBs.

Collecting data for SNMP

Debian and Ubuntu, keeping with good practice, have created an unprivileged user for running SNMP under. This is very good from a security point of view, it does mean that snmpd can't directly access any data that requires privilege on the system.

There are some options:

Using SUID programs / scripts: What this means is that a program or script may be run by anyone with access determined by the regular file permissions, but the program will be run with with the privilege of the owner, in this case root. Useful, but there is always a risk that someone malicious may discover a way to exploit a flaw and gain privilege on the system by it.
Using a CRON job: CRON can periodically run a script to collect the privileged data and dump it into a file. Scripts run by snmpd can then pick up this data without needing privilege. This means that no risky SUID stuff is needed, but does have the slight disadvantage of data being only as up to date as the last run of the CRON job.

As those who know me could predict, I chose the secure option of running a CRON job to collect the data.

I created a basic shell script in /etc/snmp/local-snmp-cronjob which is written carefully to avoid security problems (all absolute paths, careful escaping and quoting of external data etc.).

I created a custom crontab (/etc/cron.d/local-snmp) with the a single entry to run the CRON job every 5 minutes:

*/5 * * * * root /etc/snmp/local-snmp-cronjob

Many similar scripts on the web store data in /tmp or similar. This is bad! Carelessly writing files as a privileged user to /tmp or any publicly writeable areas invites symlink attacks, and indeed almost all that I looked at before deciding to roll my own where vulnerable to symlink attacks.

How a symlink attack works is that the attacker creates a symlink to some file that they want to attack (generally to damage or overwrite) and name the symlink in /tmp (or other world writeable area) matching a file that they know some program running as a privileged user will try to write. Some of these attacks rely on having to retry if the filename is only partially predictable (eg. includes the PID in the name), or exploit a race condition (eg. checking for the file before opening, rather than using O_EXCL to ensure that existing files aren't overwritten).

To avoid this, I created the directory /var/local/snmp for storing this data which is only writeable by root and so safe to use from basic scripts without careful checks.

Extending SNMP

Net-SNMP provides a simple way of adding custom extensions. This allows scripts and programs to be written which can capture just about any parameters and relay them back via SNMP. Extensions can be added by adding lines to /etc/snmp/snmpd.conf in the format:

extend NAME PATH_TO_SCRIPT OPTIONAL_SCRIPT_ARGS

The script simply dumps a load of values which may be accessed on the SNMP OID of:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."NAME".VALUENUMBER

That means that if our script with the extension name of "foo" returns the numbers:

4.3
68.9
42

Then requesting the following SNMP OIDs will give:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".1 gives 4.3
NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".2 gives 68.9
NET-SNMP-EXTEND-MIB::nsExtendOutLine."foo".3 gives 42

SNMP v3 Configuration

Unless you have good reason otherwise, I prefer using SNMP v3 to take advantage of the added security. This requires setting up two pass phrases for authentication and encryption of the data. I generally use "pwgen --secure 16" to create random 16 character pass phrases.

On the system being monitored the authentication is then set up (read only access) with the command:

$ net-snmp-config --create-snmpv3-user -ro -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS USERNAME

AUTHPROTO is the authentication hashing protocol / algorithm and may be either MD5 or SHA. PRIVPROTO is the privacy encryption protocol / algorithm and may be either DES or AES. AUTHPASS AND PRIVPASS are the authentication and privacy passwords, and USERNAME is the SNMP USERNAME for the user you are adding. This should be sufficient with the default config to monitor a host.

To test that it is working, on the host that is doing the monitoring you can connect try:

$ snmpwalk -v 3 -u USERNAME -l authPriv -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS MONITOREDHOSTNAME

If you are monitoring remotely (ie. using a MONITOREDHOSTNAME other than "localhost") then you may also need to remove the "127.0.0.1" from /etc/default/snmpd and restart snmpd to ensure that snmpd is listening on all addresses and not just on the local loopback.

You can add the OID / part of the OID after that to walk a particular section:

$ snmpwalk -v 3 -u USERNAME -l authPriv -a AUTHPROTO -x PRIVPROTO -A AUTHPASS -X PRIVPASS MONITOREDHOSTNAME NET-SNMP-EXTEND-MIB::nsExtendOutLine

SNMP v2c Configuration

v2c does not have any strong authentication or encryption of the data and as such I tend to avoid it. The basic concept is that a "community" of trusted IP addresses is created which are allowed to connect.

By default snmpd should respond to the community "public" with a subset of read-only parameters (paranoid mode). If you need to to give further access via v2c then this may be done with a combination of the "com2sec", "group", "view" and "access" parameters in /etc/snmp/snmpd.conf. The example config and manual pages have more detail on this if you need to do it. For example, to give read-only access from the local machine, add this line to the top of the "com2sec" rules:

com2sec readonly 127.0.0.1 public

To test v2c access is working, like above you can use the command:

$ snmpwalk -v 2c -c public MONITOREDHOSTNAME

Again, if you are monitoring remotely (ie. using a MONITOREDHOSTNAME other than "localhost") then you may also need to remove the "127.0.0.1" from /etc/default/snmpd and restart snmpd to ensure that snmpd is listening on all addresses and not just on the local loopback.

I am going to publish the SNMP config and Cacti templates I have created for monitoring various aspects of systems. I will list them here as I publish them:

Similar Articles

Recent Articles

Linux (Debian, Ubuntu) SNMP basics