Menu
Index

Contact
LinkedIn
GitHub
Atom Feed
Comments Atom Feed



Tweet

Similar Articles

14/05/2015 22:35
PHP APC on Cacti via SNMP
12/03/2016 15:33
PHP Zend opcache on Cacti via SNMP
08/01/2015 22:58
Nginx on Cacti via SNMP
08/01/2015 22:59
php-fpm on Cacti via SNMP
22/11/2009 16:49
Postfix stats on Cacti (via SNMP)
31/10/2009 16:02
LM Sensors stats on Cacti (via SNMP)

Recent Articles

21/03/2017 15:18
Kubernetes to learn Part 2
21/03/2017 13:53
Kubernetes to learn Part 1
17/07/2016 15:23
AWS ssh known_host sync
11/07/2016 08:42
File integrity and log anomaly auditing Updated (like fcheck/logcheck)
30/05/2016 13:09
Xenial LXC Container on Debian

Glen Pitt-Pladdy :: Blog

Apache stats on Cacti (via SNMP)

Following on from the basics of SNMP I did previously, this article now adds my next set of SNMP extension scripts, config, and Cacti templates to monitor an Apache web server.

Apache Stats

Apache fortunately generates comprehensive stats internally and presents it under the url http://localhost/server-status?auto in a form that is easily parsable by scripts. All that is needed is to extract the stats.

Because a number of hits on the server will be needed for each statistic when snmpd is queried, it is a good idea to do some basic caching so that the stats retreval doesn't become the dominant load on an otherwise lightly loaded server.

The one thing that needs to be changed in Apache's config is to switch to extended stats. On Debian and Ubuntu systems edit the file /etc/apache2/mods-available/status.conf and add the directive "ExtendedStatus on" just before the Location section. Reload apache and it should be ready to go.

Getting Apache stats over SNMP

As the stats are available to the local machine (localhost or 127.0.0.1) over http, we once again avoid acrobatics discussed previously needed to get some data into snmpd. A single extension script is all that is needed.

To allow caching, I have created an additional directory: /var/local/snmp/cache

Set this to be owned by snmp so that it can write and modify files in here: chown snmp: /var/local/snmp/cache

Download: Apache statistics SNMP scripts and Cacti Templates are on GitHub

There is a choice between a Python script apache-stats.py and a Shell script apache-stats.sh. The shell script is likely to have more overhead due to spinning up a load of processes each time it gets hit. With Python you will also need the python-urlgrabber package (or at least that's what it's called in Debian / Ubuntu), and with the Shell option you will need wget.

I place this script (make it executable first: chmod +x apache-stats.py) in /etc/snmp

In /etc/snmp/snmpd.conf add the following line (or others if you want to monitor them):

extend apache /etc/snmp/apache-stats.py

.... or as appropriate if you are using the shell version

Once you have added all this in you can test apache-stats by running it from the command line with appropriate parameters, and via SNMP by appending the appropriate SNMP OID to the "snmpwalk" commands shown previously.

Cacti Templates

I have generated some basic Cacti Templates for these Apache stats.

Simply import the template cacti_host_template_apache.xml, and add the graphs you want to the appropriate device graphs in Cacti. It should just work if your SNMP is working correctly for that device (ensure other SNMP parameters are working for that device).

Graph Screen Shots

Apache Stats over SNMP on Cacti : Bytes per Request

Apache Stats over SNMP on Cacti : Bytes per Second

Apache Stats over SNMP on Cacti : CPU Load

Apache Stats over SNMP on Cacti : Scoreboard

Apache Stats over SNMP on Cacti : Workers

Update:

One person had pointed out that there has been some problems on some versions of Cacti with importing the templates. The template was generated with version 0.8.7b (from Debain Lenny).

The problem manifests it's self as "Cacti version does not exist" error, and appears to be cured by adding in this version, although in my version the file is actually global_arrays.php

The relevant array from my global_arrays.php / config_array.php:

$hash_version_codes = array(
    "0.8.4"  => "0000",
    "0.8.5"  => "0001",
    "0.8.5a" => "0002",
    "0.8.6"  => "0003",
    "0.8.6a" => "0004",
    "0.8.6b" => "0005",
    "0.8.6c" => "0006",
    "0.8.6d" => "0007",
    "0.8.6e" => "0008",
    "0.8.6f" => "0009",
    "0.8.6g" => "0010",
    "0.8.6h" => "0011",
    "0.8.6i" => "0012",
    "0.8.6j" => "0013",
    "0.8.7"  => "0014",
    "0.8.7a" => "0015",
    "0.8.7b" => "0016",
    "0.8.7c" => "0017",
    "0.8.7d" => "0018",
    "0.8.7e" => "0019",
    "0.8.7f" => "0020",
    "0.8.7g" => "0021"
    );

Comments:

Ajay Kajla Image  19/05/2010 04:25 :: Ajay Kajla

Hi,

The python script is giving following error. Can you please check.

./apache-stats
16
9216
Traceback (most recent call last):
File "./apache-stats.1", line 78, in ?
print params[param]
KeyError: 'CPULoad'
[root@ajay snmp]# ./apache-stats
16
9216
Traceback (most recent call last):
File "./apache-stats", line 78, in ?
print params[param]
KeyError: 'CPULoad'

Glen Pitt-Pladdy Image  19/05/2010 07:08 :: Glen Pitt-Pladdy

Thanks for pointing this out. Indeed, on some platforms (64bit?) Apache seems not to have a CPULoad parameter.

I have updated the script to handle missing parameters so hopefully this will solve the problem for you.

Rob Moser Image  20/05/2010 17:50 :: Rob Moser

Hi Glen,

Doing something very similar today, and this was a great help; thanks.

I had two problems implementing this.  1) My installation of SNMP didn't have the MIB for NET-SNMP-EXTEND-MIB installed.  Not your fault, obviously, but I thought I'd mention it here in case someone else following your advice has the same problem.  If you test the SNMP extension with snmpwalk and get an error like "No Such Instance currently exists at this OID", have a look at the instructions for adding MIBs at www.net-snmp.org/docs/FAQ.html#How_do_I_add_a_MIB_

Problem 2 was that the graph and data templates for Bytes per Request didn't seem to import properly.  It seemed to have something to do with the fact that it was using bytesPRErequest instead of bytesPERrequest (emphasis mine) in some places; I changed them all to PER, deleted all of the graphs and data sources and re-built them from the new templates, and it seems to be working fine.

Glen Pitt-Pladdy Image  20/05/2010 18:27 :: Glen Pitt-Pladdy

Hi Rob, Thanks for the feedback.

Interesting that your net-snmp didn't have that MIB. On my Debian installs they all did. The link you provided should hopefully do the trick for anyone else with this problem.

I think it's interesting that the pre/per thing caused problems. It was a typo that propagated from the Data template to the Graph template. I have to admit that I have had the occasional problem with templates with Cacti and generally clearing them out and re-importing seems to do the trick for me.

I have put up an updated version of the template with this corrected. Hopefully that will avoid any others with these problems.

Thanks!

Rob Moser Image  20/05/2010 18:31 :: Rob Moser

I thought it was odd too; looked like it was done consistently throughout, so it shouldn't have bothered anyone but pedants like me.  But of course I only really started looking at it after I had half fixed it, so I didn't have it in the original state to check.  Its entirely possible that the typo had nothing to do with it, and re-loading things in Cacti is the only thing that I needed to fix my problem.

Thanks again for the handy walkthrough!

Rob Moser Image  21/05/2010 23:03 :: Rob Moser

Turned out one of our machines didn't have python (and we don't want it to) so I ported this to perl.  Saved a copy at docs.google.com/Doc?docid=0AT8aSke8tPgGZHZ2N3Bydl8xcHFtcHRzZGs&hl=en if its of any use to you...

Glen Pitt-Pladdy Image  23/05/2010 15:45 :: Glen Pitt-Pladdy

Nicely done!

I have to confess that doing stuff in Python is just an excuse for me to learn it  :)

If anyone is unclear on how to get Rob's script working - you will need to add a #!/usr/bin/perl (or wherever your perl lives) at the top or modify your snmpd.conf to run this file with perl. Other than that, it looks the business to me and should be a drop-in replacement if you don't want to mess with Python.

kirt Image  07/11/2010 05:27 :: kirt

In Data Templates MIB should be liested as follows:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".17

Note quotes around "apache". Net-SNMP refuses to fetch data without that quotes.

Glen Pitt-Pladdy Image  07/11/2010 09:51 :: Glen Pitt-Pladdy

I may be missing something here, but the template currently appears to already have the quotes. Could you give me more details exactly where I can find this (eg. line number in the template).

Andreas Laub Image  09/02/2011 10:28 :: Andreas Laub

Hi,

I used your plugin in Debian Lenny, now I upgraded to squeeze. The cacti Version is now 0.8.7g-1. The Problem is, that no data is written to the graphs. Before the Upgrade everything works well!

These Messages I get in the cacti.log.

02/09/2011 11:25:05 AM - CMDPHP: Poller[0] Host[23] DS[638] WARNING: Result from SNMP not valid.  Partial Result: U
02/09/2011 11:25:05 AM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'212.21.168.4', and OID:'NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache".13'

Please help me to find a solution.

Thanks.

Best Regards
Andreas

Glen Pitt-Pladdy Image  09/02/2011 11:31 :: Glen Pitt-Pladdy

The first place I would look are for replqced config files. The most obvious ones are /etc/snmp/snmpd.conf where the config above may have been replaced, and /etc/default/snmpd which may now be only listening locally.

Failing that, I would suggest checking over the config and doing snmpwalk step-by-step to find where things break.

I have not upgraded any web servers yet so don't know of anything squeeze specific that may break things.

Andreas Laub Image  09/02/2011 11:50 :: Andreas Laub

Hi!

I found out what's wrong. You have to install the package snmp-mibs-downloader from the non-free repository and commend the mibs line in /etc/snmp/snmp.conf:
#mibs :
after doing that it works well again :-)

Thanks for your help!

Sage Image  26/03/2011 15:27 :: Sage

Nice script and instructions.  I have a very low traffic site I'm trying this on and it would appear the RequestsPreSecond aren't meeting the non Bandwidth instructions.  I'm seeing a vertical label of M and the number being multiplied by the Bytes * 1024.  Should be reading more like 1rps and not 10M.

Let me know if can provide more info on versions/OS/etc...

Glen Pitt-Pladdy Image  26/03/2011 15:50 :: Glen Pitt-Pladdy

Thanks! Nice to hear people are finding the scripts and templates useful.

There is a multiply by 1024 to get the Total traffic in bytes, but I don't believe I graph that - it's just there for completeness. If your run "wget -O- http://localhost/server-status?auto" it will give you the stats from the Apache server, and these should be the same as what's being graphed.

One thing to consider is that if you are getting very low levels of traffic then the dominant traffic could actually be the stats being polled. Also beware of any other automated things that may be hitting your Apache, or any other sites / proxys or other things which Apache may be doing. eg. Checking CCTV video with Zoneminder's web interface totally dwarfs the volume of traffic this blog generates.

For reference, I'm mostly running Debian and Ubuntu across many versions on the various servers that this monitoring is deployed.

Sage Image  26/03/2011 19:27 :: Sage

wget returns this
otal Accesses: 3831
Total kBytes: 36578
CPULoad: .206492
Uptime: 97442
ReqPerSec: .0393157
BytesPerSec: 384.391
BytesPerReq: 9777.05
BusyWorkers: 1
IdleWorkers: 13
Scoreboard: ____________W_.....................................................
...............................................................................
...............................................................................
...............................
100%[======================================>] 434         --.-K/s   in 0s

Here is a link to my graphs.... guest/guest http://sandbox.mindcrime.net/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=38">http://sandbox.mindcrime.net/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=38">http://sandbox.mindcrime.net/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=38

Glen Pitt-Pladdy Image  26/03/2011 19:52 :: Glen Pitt-Pladdy

Hmmmm.... does seem rather odd, however I suspect there is something broken or giving invalid data somewhere along the way, and ~15M is the limit so it's bouncing around the limit.

Apache is clearly giving valid data, so let's follow the chain through.  Next, execute the stats script as the snmp user - you may have to give the snmp user a valid shell in /etc/passwd, but remember to change it back afterwards:

$ sudo su - snmp -c /etc/snmp/apache-stats

This should print out a stack of data for snmpd to use. Requests per second should be the 5th item. If it's giving bad stats then the problem is with the script. Also check the cache file in /var/local/snmp/cache/apache which should match what you got from wget.

If that's fine then do an snmpwalk from the monitoring machine and see if snmpd is mangling things in some way.

Somewhere down the chain something goes wrong so let's find the point where the problem occurs as a first step.

Sage Image  26/03/2011 20:42 :: Sage

SNMP is run by root by default on my distro

[root@sandbox mibs]# cat /etc/redhat-release
CentOS release 5.5 (Final)
[root@sandbox mibs]# uname -a
Linux sandbox.mindcrime.net 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@sandbox mibs]# ps auwx | grep snmp
root      5942  0.0  0.5 160684 11224 ?        Sl   Mar25   0:26 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a

the cache indeed matches the wget

I have the NET-SNMP-EXTEND-MIB.txt listed in my mib dir, and once I find the mib/oid string I'll include it's output.  The site is brand new and doubtful it's hitting any limit of any kind.

Thanks for the quick replies!  Nice support

Sage Image  26/03/2011 21:30 :: Sage

I hard set the OID and it appears to have one extra value I think throwing it off...may be due to my adjustments
snmpwalk

UCD-SNMP-MIB::ucdavis.51.1.1 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.51.2.1 = STRING: "apache"
UCD-SNMP-MIB::ucdavis.51.3.1 = STRING: "/etc/snmp/apache-stats"
UCD-SNMP-MIB::ucdavis.51.100.1 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.51.101.1 = STRING: "4156"
UCD-SNMP-MIB::ucdavis.51.101.2 = STRING: "41772032"
UCD-SNMP-MIB::ucdavis.51.101.3 = STRING: ".208511"
UCD-SNMP-MIB::ucdavis.51.101.4 = STRING: "104973"
UCD-SNMP-MIB::ucdavis.51.101.5 = STRING: ".0395911"
UCD-SNMP-MIB::ucdavis.51.101.6 = STRING: "397.931"
UCD-SNMP-MIB::ucdavis.51.101.7 = STRING: "10051"

Glen Pitt-Pladdy Image  27/03/2011 10:43 :: Glen Pitt-Pladdy

Assuming the adjustments you are making are not affecting the numbering, from what you have shown, "5" is still the Requests per second.

This suggests that the problem is actually happening in Cacti - possibly the template has imported wrongly or something similar.  Some extra checks:

* In Cacti, go to the data template for Requests Per Second and check the Data Source Type - it should be on Guage (ie. just take the number as it comes)

* Go to the actual data source for this host and verify it is using the correct Data template.

* Go to the graph for this host and verify it is using the correct graph template and data source.

* Then find where the rrd files are kept (/var/lib/cacti/rra/ on Debian) and find the file for this data source. Run "rrdtool info NAME_OF_FILE.rrd" and note in particular the ds[apache_reqpersec].type which should once again be GUAGE. Also note the ds[apache_reqpersec].last_ds which should match the number handed out by Apache and snmpd.

Mxx Image  12/04/2011 21:32 :: Mxx

Great script. I'm looking to implement in my install.
Why do you run the other part of the script from cron? Why not collect apache stats when snmp request actually hits?

Glen Pitt-Pladdy Image  12/04/2011 23:00 :: Glen Pitt-Pladdy

I think a little confusion has developed here. While my general article on snmp describes using cron to run a data gathering script to gather privileged info (eg. check SMART parameters for disks or parse a postfix log file), the config described in this article does exactly what you describe and runs the data gathering script directly from snmpd.

Nils Image  21/09/2015 13:43 :: Nils

Hi,

thanks for your scripts and templates, they are very useful !

I had some difficulties with the python script lately, it stopped working, throwing an error about fields[1] having a list index out of range on line 63. I noticed my cache file started with a 'localhost' line (nothing more, but all the other lines have multiple fields). I guessed that line was causing the index problem, so I changed line 61 from "else:" to "elif fields[0] != 'localhost':" and got the script working again.

I'm running the script on a NetBSD 6.1.x/amd64 system with Apache 2.4.x and Python 2.7.x

I hope it will help someone !

Christian Image  29/11/2016 12:41 :: Christian

I'm getting error "./apache-stats.sh: line 47: *1024: syntax error: operand expected (error token is "*1024")" when trying run ./apache-stats.sh in my web server, please your advice for this.Thanks

Glen Pitt-Pladdy Image  30/11/2016 07:22 :: Glen Pitt-Pladdy

The line is a simple multiply so what is likely is that either the stats aren't being retrieved (try pulling the stats URL with curl or wget and examine it), are in a format the script can't parse or the cache file has a problem (eg. some permissions problem writing the file when run as the user snmpd runs it as). Check those and we'll see if we need to dig deeper.

andhikagg Image  17/12/2016 16:36 :: andhikagg

Hi Glen Pitt-Pladdy, i've problem with -nan graph value, below for details :

Data Source Debug :
/usr/bin/rrdtool create \
/var/lib/cacti/rra/research_example_com_apache_accesses_956.rrd \
--step 60  \
DS:apache_accesses:COUNTER:120:0:1000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

/usr/bin/rrdtool create \
/var/lib/cacti/rra/research_example_com_apache_bytes_957.rrd \
--step 60  \
DS:apache_bytes:COUNTER:120:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

andhikagg Image  17/12/2016 16:41 :: andhikagg

--continue--

Additional Directory for Cache :
/var/local/snmp/cache/apache (exist) with permission: 644 and owner: snmp:snmp

Content /var/local/snmp/cache/apache :
Total Accesses: 28840
Total kBytes: 193957
CPULoad: .0105857
Uptime: 205750
ReqPerSec: .14017
BytesPerSec: 965.307
BytesPerReq: 6886.68
BusyWorkers: 1
IdleWorkers: 9
Scoreboard: ____.__W___..........................................................................................................................................

Result apache-stats.py :
28840
198611968
.0105857
205750
.14017
965.307
6886.68
1
9
9
0
0
1
0
0
0
0
0
0
140

Note :
- Ubuntu 14.04.4 LTS
- Cacti Version : 0.8.8b
- PHP : 5.5.9
- Web Server : 2.4.7

Glen Pitt-Pladdy Image  17/12/2016 16:51 :: Glen Pitt-Pladdy

Please don't post large amounts of outputs of tools in future.

This shows that the Apache status data is working on the host, but not that anything is working between the collection script and Cacti. Take a look at my other article on Debugging Cacti Problems and the article on SNMP basics where you should test that you can retrieve the data with snmpwalk. Those should allow you to isolate the problem.

andhikagg Image  18/12/2016 08:36 :: andhikagg

Hi Glen Pitt-Pladdy, I'm sorry about that.

I've run : snmpwalk -v 2c -c public 10.0.0.213 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."apache"' and it get value like my post above.

I didn't see clue to solving my problem from your article https://www.pitt-pladdy.com/blog/_20151108-193119_0000_Debugging_Cacti_Problems/, can you help me step by step ?

Glen Pitt-Pladdy Image  20/12/2016 07:20 :: Glen Pitt-Pladdy

So, assuming you're running snmpwalk from the Cacti server, that verifies that the data is being collected and made available - ie. the Apache config, extension script and snmpd are all good. That suggests the problem is most likely Cacti it's self. So as per the debugging article, check the Cacti-side config, and if needed turn up the logging and check what's going on on that side.




Are you human? (reduces spam)
Note: Identity details will be stored in a cookie. Posts may not appear immediately