Glen Pitt-Pladdy :: Blog

OpenVz has a very comprehensive set of resource limits which can have severe impact on processes within the container when they are hit. Basically, many programs fail badly when they are not given the resources they expect. It's important to monitor the limits and know when they fail. It's also very useful to graph them so that you can see trends and take pre-emptive action.

The resource allocation status in OpenVz is provided in /proc/user_beancounters. Cacti being my weapon of choice for graphing (and plugins Cacti can even give you early warnings of approaching limits), I built a set of templates for OpenVz user beancounters.

Collecting the data

As /proc/user_beancounters is only readable by root we use a cron job as described in SNMP Basics to collect this data. Assuming you put the data collection script in /etc/snmp and the stats file in /var/local/snmp then you can run the collection with something like:

# run openvz checks
/etc/snmp/vzbeancheck /var/local/snmp/vz

Each time it is run it will update the stats file and if it gets a line with new failures (limits breached) on it then it will also output a summary. This means that failures get reported via Cron. If you don't want that then just send the output to /dev/null

The collection script also looks under your OpenVz directory and reads /etc/hostname and /etc/hosts for each container so that it can use the full hostname of the container. If your containers are stored in a different location to /var/lib/vz then you will need to modify the script to match your configuration.

It's also worth noting that the data in the stats file is just a different representation of /proc/user_beancounters so if that data is sensitive then you need to make provision for restricting access to the stats file with appropriate permissions.

Download: OpenVz stats collection script (Perl)

Extending snmpd

There are lots of stats for OpenVz. Each parameter has the currently held amount, the maximum held amount, the barrier, the limit and a count of how many failures since the container was started. All together that's 101 individual values collected for each container.

As a result I have produced a generic extension script for getting the data into snmpd and I put this in /etc/snmp

Download: OpenVz snmpd extension script (Perl)

As a result there are a lot of lines to add to /etc/snmpd.conf for all the parameters that are used:

extend vzhostnames /etc/snmp/vz-generic hostname
extend vzkmemsizeheld /etc/snmp/vz-generic kmemsize:held
extend vzkmemsizemaxheld /etc/snmp/vz-generic kmemsize:maxheld
extend vzkmemsizebarrier /etc/snmp/vz-generic kmemsize:barrier
extend vzkmemsizelimit /etc/snmp/vz-generic kmemsize:limit
extend vzkmemsizefailcnt /etc/snmp/vz-generic kmemsize:failcnt
extend vzlockedpagesheld /etc/snmp/vz-generic lockedpages:held
extend vzlockedpagesmaxheld /etc/snmp/vz-generic lockedpages:maxheld
extend vzlockedpagesbarrier /etc/snmp/vz-generic lockedpages:barrier
extend vzlockedpageslimit /etc/snmp/vz-generic lockedpages:limit
extend vzlockedpagesfailcnt /etc/snmp/vz-generic lockedpages:failcnt
extend vzprivvmpagesheld /etc/snmp/vz-generic privvmpages:held
extend vzprivvmpagesmaxheld /etc/snmp/vz-generic privvmpages:maxheld
extend vzprivvmpagesbarrier /etc/snmp/vz-generic privvmpages:barrier
extend vzprivvmpageslimit /etc/snmp/vz-generic privvmpages:limit
extend vzprivvmpagesfailcnt /etc/snmp/vz-generic privvmpages:failcnt
extend vzshmpagesheld /etc/snmp/vz-generic shmpages:held
extend vzshmpagesmaxheld /etc/snmp/vz-generic shmpages:maxheld
extend vzshmpagesbarrier /etc/snmp/vz-generic shmpages:barrier
extend vzshmpageslimit /etc/snmp/vz-generic shmpages:limit
extend vzshmpagesfailcnt /etc/snmp/vz-generic shmpages:failcnt
extend vznumprocheld /etc/snmp/vz-generic numproc:held
extend vznumprocmaxheld /etc/snmp/vz-generic numproc:maxheld
extend vznumprocbarrier /etc/snmp/vz-generic numproc:barrier
extend vznumproclimit /etc/snmp/vz-generic numproc:limit
extend vznumprocfailcnt /etc/snmp/vz-generic numproc:failcnt
extend vzphyspagesheld /etc/snmp/vz-generic physpages:held
extend vzphyspagesmaxheld /etc/snmp/vz-generic physpages:maxheld
extend vzphyspagesbarrier /etc/snmp/vz-generic physpages:barrier
extend vzphyspageslimit /etc/snmp/vz-generic physpages:limit
extend vzphyspagesfailcnt /etc/snmp/vz-generic physpages:failcnt
extend vzvmguarpagesheld /etc/snmp/vz-generic vmguarpages:held
extend vzvmguarpagesmaxheld /etc/snmp/vz-generic vmguarpages:maxheld
extend vzvmguarpagesbarrier /etc/snmp/vz-generic vmguarpages:barrier
extend vzvmguarpageslimit /etc/snmp/vz-generic vmguarpages:limit
extend vzvmguarpagesfailcnt /etc/snmp/vz-generic vmguarpages:failcnt
extend vzoomguarpagesheld /etc/snmp/vz-generic oomguarpages:held
extend vzoomguarpagesmaxheld /etc/snmp/vz-generic oomguarpages:maxheld
extend vzoomguarpagesbarrier /etc/snmp/vz-generic oomguarpages:barrier
extend vzoomguarpageslimit /etc/snmp/vz-generic oomguarpages:limit
extend vzoomguarpagesfailcnt /etc/snmp/vz-generic oomguarpages:failcnt
extend vznumtcpsockheld /etc/snmp/vz-generic numtcpsock:held
extend vznumtcpsockmaxheld /etc/snmp/vz-generic numtcpsock:maxheld
extend vznumtcpsockbarrier /etc/snmp/vz-generic numtcpsock:barrier
extend vznumtcpsocklimit /etc/snmp/vz-generic numtcpsock:limit
extend vznumtcpsockfailcnt /etc/snmp/vz-generic numtcpsock:failcnt
extend vznumflockheld /etc/snmp/vz-generic numflock:held
extend vznumflockmaxheld /etc/snmp/vz-generic numflock:maxheld
extend vznumflockbarrier /etc/snmp/vz-generic numflock:barrier
extend vznumflocklimit /etc/snmp/vz-generic numflock:limit
extend vznumflockfailcnt /etc/snmp/vz-generic numflock:failcnt
extend vznumptyheld /etc/snmp/vz-generic numpty:held
extend vznumptymaxheld /etc/snmp/vz-generic numpty:maxheld
extend vznumptybarrier /etc/snmp/vz-generic numpty:barrier
extend vznumptylimit /etc/snmp/vz-generic numpty:limit
extend vznumptyfailcnt /etc/snmp/vz-generic numpty:failcnt
extend vznumsiginfoheld /etc/snmp/vz-generic numsiginfo:held
extend vznumsiginfomaxheld /etc/snmp/vz-generic numsiginfo:maxheld
extend vznumsiginfobarrier /etc/snmp/vz-generic numsiginfo:barrier
extend vznumsiginfolimit /etc/snmp/vz-generic numsiginfo:limit
extend vznumsiginfofailcnt /etc/snmp/vz-generic numsiginfo:failcnt
extend vztcpsndbufheld /etc/snmp/vz-generic tcpsndbuf:held
extend vztcpsndbufmaxheld /etc/snmp/vz-generic tcpsndbuf:maxheld
extend vztcpsndbufbarrier /etc/snmp/vz-generic tcpsndbuf:barrier
extend vztcpsndbuflimit /etc/snmp/vz-generic tcpsndbuf:limit
extend vztcpsndbuffailcnt /etc/snmp/vz-generic tcpsndbuf:failcnt
extend vztcprcvbufheld /etc/snmp/vz-generic tcprcvbuf:held
extend vztcprcvbufmaxheld /etc/snmp/vz-generic tcprcvbuf:maxheld
extend vztcprcvbufbarrier /etc/snmp/vz-generic tcprcvbuf:barrier
extend vztcprcvbuflimit /etc/snmp/vz-generic tcprcvbuf:limit
extend vztcprcvbuffailcnt /etc/snmp/vz-generic tcprcvbuf:failcnt
extend vzothersockbufheld /etc/snmp/vz-generic othersockbuf:held
extend vzothersockbufmaxheld /etc/snmp/vz-generic othersockbuf:maxheld
extend vzothersockbufbarrier /etc/snmp/vz-generic othersockbuf:barrier
extend vzothersockbuflimit /etc/snmp/vz-generic othersockbuf:limit
extend vzothersockbuffailcnt /etc/snmp/vz-generic othersockbuf:failcnt
extend vzdgramrcvbufheld /etc/snmp/vz-generic dgramrcvbuf:held
extend vzdgramrcvbufmaxheld /etc/snmp/vz-generic dgramrcvbuf:maxheld
extend vzdgramrcvbufbarrier /etc/snmp/vz-generic dgramrcvbuf:barrier
extend vzdgramrcvbuflimit /etc/snmp/vz-generic dgramrcvbuf:limit
extend vzdgramrcvbuffailcnt /etc/snmp/vz-generic dgramrcvbuf:failcnt
extend vznumothersockheld /etc/snmp/vz-generic numothersock:held
extend vznumothersockmaxheld /etc/snmp/vz-generic numothersock:maxheld
extend vznumothersockbarrier /etc/snmp/vz-generic numothersock:barrier
extend vznumothersocklimit /etc/snmp/vz-generic numothersock:limit
extend vznumothersockfailcnt /etc/snmp/vz-generic numothersock:failcnt
extend vzdcachesizeheld /etc/snmp/vz-generic dcachesize:held
extend vzdcachesizemaxheld /etc/snmp/vz-generic dcachesize:maxheld
extend vzdcachesizebarrier /etc/snmp/vz-generic dcachesize:barrier
extend vzdcachesizelimit /etc/snmp/vz-generic dcachesize:limit
extend vzdcachesizefailcnt /etc/snmp/vz-generic dcachesize:failcnt
extend vznumfileheld /etc/snmp/vz-generic numfile:held
extend vznumfilemaxheld /etc/snmp/vz-generic numfile:maxheld
extend vznumfilebarrier /etc/snmp/vz-generic numfile:barrier
extend vznumfilelimit /etc/snmp/vz-generic numfile:limit
extend vznumfilefailcnt /etc/snmp/vz-generic numfile:failcnt
extend vznumiptentheld /etc/snmp/vz-generic numiptent:held
extend vznumiptentmaxheld /etc/snmp/vz-generic numiptent:maxheld
extend vznumiptentbarrier /etc/snmp/vz-generic numiptent:barrier
extend vznumiptentlimit /etc/snmp/vz-generic numiptent:limit
extend vznumiptentfailcnt /etc/snmp/vz-generic numiptent:failcnt

Restart snmpd and you should have the paramers available via SNMP.

Indexed SNMP with Cacti

Cacti requires an XML file describing how to map the SNMP data to each container. As this is a local (unpackaged) version I have done my configuration around putting this file in /usr/local/share/cacti/resource/snmp_queries/ and you will need to alter the templates if you put the file elsewhere.

Download: OpenVz Cacti SNMP Query (XML)

Put this in /usr/local/share/cacti/resource/snmp_queries/ or wherever appropriate for your system.

Next, import the Cacti Template.

Download: OpenVz Cacti Template

To use the graphs add the OpenVz data query to each OpenVz host and then go to the Create Graphs page and all going to plan you should see a list of Containers (by hostname) and possible graphs to create for that container.