Menu
Index

Contact
LinkedIn
GitHub
Atom Feed
Comments Atom Feed



Tweet

Similar Articles

31/10/2009 14:46
SMART stats on Cacti (via SNMP)
31/10/2009 11:03
Linux (Debian, Ubuntu) SNMP basics
16/11/2011 20:12
OpenVz User Beancounters (UBC) on Cacti via SNMP
14/05/2015 22:35
PHP APC on Cacti via SNMP
22/11/2009 16:49
Postfix stats on Cacti (via SNMP)
14/11/2009 13:46
Apache stats on Cacti (via SNMP)

Recent Articles

21/03/2017 15:18
Kubernetes to learn Part 2
21/03/2017 13:53
Kubernetes to learn Part 1
17/07/2016 15:23
AWS ssh known_host sync
11/07/2016 08:42
File integrity and log anomaly auditing Updated (like fcheck/logcheck)
30/05/2016 13:09
Xenial LXC Container on Debian

Glen Pitt-Pladdy :: Blog

Dovecot stats on Cacti (via SNMP)

As promised in the comments about the Postfix stats article, this posting kicks off support for Dovecot stats with Cacti, extending the original Postfix only stats.

This uses my Universal Log Analyser providing a plugin for Dovecot. You will need to get this installed and working before you can use the plugin here, and have it running from a cron job every 5 minutes (or whatever your sample times is for Cacti) so that it can parse the logs.

At this stage I am only producing stats for IMAP, but with the nice consistent log format it should be easy to add POP3 and SIEVE later.

Dovecot Plugin

Throw dovecot.pm in your plugin directory for the Universal Log Analyser, and add "dovecot" as a module to the command line so that this module gets loaded.

Download: Universal Log Analyser plugin and Cacti template for Dovecot are on GitHub

Not much more to it than that. The remainder of the article assumes your stats file is /var/local/snmp/mail so if it isn't you will need to tweak things to match your install.

It's worth checking the stats file to verify that dovecot stats are in fact being picked up by the plugin.

SNMP Scripts

First, ensure that your SNMP is configured and working as described in my SNMP basics article.

These provide the link for snmpd to pick up the stats and assuming they are in /etc/snmp the config in /etc/snmp/snmpd.conf is:

extend dovecotauth /etc/snmp/dovecot-stats-auth
extend dovecotdeliver /etc/snmp/dovecot-stats-deliver
extend dovecotimapcrypto /etc/snmp/dovecot-stats-imap-crypto
extend dovecotimapdata /etc/snmp/dovecot-stats-imap-data
extend dovecotimapdisconnect /etc/snmp/dovecot-stats-imap-disconnect
extend dovecotimaplogin /etc/snmp/dovecot-stats-imap-login
extend dovecotimaploginmethod /etc/snmp/dovecot-stats-imap-loginmethod
extend dovecotimaploginmethod /etc/snmp/dovecot-stats-imap-loginmethod
extend dovecotpop3crypto /etc/snmp/dovecot-stats-pop3-crypto
extend dovecotpop3data /etc/snmp/dovecot-stats-pop3-data
extend dovecotpop3disconnect /etc/snmp/dovecot-stats-pop3-disconnect
extend dovecotpop3login /etc/snmp/dovecot-stats-pop3-login
extend dovecotpop3loginmethod /etc/snmp/dovecot-stats-pop3-loginmethod
extend dovecotpop3loginmethod /etc/snmp/dovecot-stats-pop3-loginmethod
extend dovecotmanagesievecrypto /etc/snmp/dovecot-stats-managesieve-crypto
extend dovecotmanagesievedata /etc/snmp/dovecot-stats-managesieve-data
extend dovecotmanagesiedisconnect /etc/snmp/dovecot-stats-managesieve-disconnect
extend dovecotmanagesielogin /etc/snmp/dovecot-stats-managesieve-login
extend dovecotmanagesieloginmethod /etc/snmp/dovecot-stats-managesieve-loginmethod
extend dovecotmanagesieloginmethod /etc/snmp/dovecot-stats-managesieve-loginmethod
extend dovecotsessions /etc/snmp/dovecot-stats-sessions

Extension scripts for snmpd are named dovecot-stats-* - put them in a suitable place.... like /etc/snmp

You should be able to run these scripts manually and they should spit back the current info from the stats file. Remember to restart snmpd so that the new config is picked up and we should be ready to go.

Cacti Template

Import this into your Cacti and add graphs as usual.

After that, assuming everything is working then after a couple data samples content should start to appear on the graphs.

If not then check the data at each step: the stats file, SNMP scripts, snmpwalk from the Cacti server, check Cacti Poller log for errors, and try Cacti in debug mode for graphs and data sources to see if that shows anything.

I will post example graphs once my ones are mature enough to have some useful data on them.

The Graphs

Dovecot Crytogrphy Stats with Cacti

Dovecot Data Stats with Cacti

Dovecot Disconnect Stats with Cacti

Dovecot Login Stats with Cacti

Dovecot Login Stats with Cacti

Dovecot Auth Error Stats with Cacti

Dovecot Delivery Stats with Cacti

Comments:

Glen Pitt-Pladdy Image  30/08/2011 07:15 :: Glen Pitt-Pladdy

See some comments on the SNMP basics post regarding tweaking this plugin for different format (dedicated) logs on different systems.

Daniele Palumbo Image  08/02/2012 17:12 :: Daniele Palumbo

hi,

i have this errors:
unknown dovecot: Feb  5 04:03:00 mail dovecot: POP3(xxx@domain.it): Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0
unknown dovecot: Feb  8 15:22:58 mail dovecot: pop3-login: Aborted login (auth failed, 1 attempts): user=<postmaster@domain.it>, method=PLAIN, rip=XX.XXX.XXX.X, lip=XXX.XXX.X.XX

the version is:
20110626 (the last i guess).

thanks,
Daniele

Glen Pitt-Pladdy Image  08/02/2012 19:58 :: Glen Pitt-Pladdy

Thanks for the report - I had not tested pop3 on a live setup as my servers only run imap so it is really useful to get these reports. I have released a new version tonight which should understand these lines and adds an additional "login Aborted" graph.

Let me know if you discover any further lines it does not understand.

Daniele Palumbo Image  09/02/2012 15:48 :: Daniele Palumbo

Hi again,

seems only that two errors for dovecot (i will look later at postfix, but anyway strings does not go to stderr).

Instead, i've found a bug in the template, it look for "dovecotpop3isconnect" instead of "dovecotpop3disconnect".
I have already patched the file, but how can i send it to you?

Thanks,
Daniele

Glen Pitt-Pladdy Image  09/02/2012 16:08 :: Glen Pitt-Pladdy

Good to hear that the scripts are working better.

Cacti can be have problems re-importing templates so it's best I make the changes in cacti and re-export the templates. I have found the typos you spotted (well done!) and have released a new template. The new template should have a MD5 checksum of 6e8f655721539da3d17c2a7f22e0b23d - if you keep getting the old one then you may need to force reload without cache (shift-F5 in many browsers).

Alex Image  25/07/2012 17:42 :: Alex

Jul 25 18:22:03 XXXXX dovecot: auth-worker(default): mysql: Connected to localhost (XXXXX)

I don't speak enough PERL for adding the needed code. :(
Can you build an update, please?

Mny Tnx
Alex

Glen Pitt-Pladdy Image  26/07/2012 15:48 :: Glen Pitt-Pladdy

Thanks for submitting the unhandled line. I've just released version 20120726 which should handle those lines.

Tomek Image  03/06/2013 19:13 :: Tomek

Hi, I have one more unhandled line:
Jun  3 20:38:14 chaos dovecot: pop3-login: Login: user=<tomek>, method=PLAIN, rip=192.168.0.135, lip=xx.xx.xx.xx, mpid=24423, TLS, session=<ur1xRETeEADAqACH>
Please help :)

Glen Pitt-Pladdy Image  22/06/2013 13:56 :: Glen Pitt-Pladdy

I think these relate to newer versions of dovecot (eg. in Debian wheezy) and with starting to upgrade my servers I'm starting to see similar. I will release an update shortly to handle the new formats/info in these lines and others. Busy testing it right now....

Alex Image  24/06/2013 07:04 :: Alex

Hi, I have another unhandled line:

/etc/snmp/uloganalyser-plugin/dovecot.pm 20120501:192 /var/log/mail.log:7842 unknown dovecot: Jun 24 08:45:03 XXXXXXX dovecot: deliver(XX@XXX.XX): sieve: msgid=<XX@XXX.XX>: forwarded to <xx@xxx.xx>

Many thanks for your work :)

Vlad Image  20/09/2013 11:39 :: Vlad

First of all, thanks for your great work! Below I have a few more examples of errors:
dovecot.pm 20121115:157 /var/log/mail_test.log:489 unknown dovecot: Sep 15 06:28:45 mc1 dovecot: imap-login: Login: user=<xxxxxx@xxxxxx>, method=PLAIN, rip=1.1.1.1, lip=1.1.1.1, mpid=27826, secured, session=<ROj3hWTmSQCsFtyb>
dovecot.pm 20121115:157 /var/log/mail_test.log:471 unknown dovecot: Sep 15 06:28:45 mc1 dovecot: pop3-login: Login: user=<xxxxx@xxxxxx>, method=PLAIN, rip=1.1.1.1, lip=1.1.1.1, mpid=27833, secured, session=<J0n3hWTmMACsFtyb>
dovecot.pm 20121115:157 /var/log/mail_test.log:389 unknown dovecot: Sep 15 06:28:01 mc1 dovecot: managesieve-login: Login: user=<xxxxx@xxxxxx>, method=PLAIN, rip=1.1.1.1, lip=1.1.1.1, mpid=27835, secured, session=<UUFQg2Tm8ACsFtya>

Glen Pitt-Pladdy Image  21/09/2013 07:26 :: Glen Pitt-Pladdy

Actually this is all covered in my development version (been updating it for Wheezy) so I've done some extra testing on that and released that today. It should cover a whole load of things in newer versions of Dovecot. The main changes are dovecot.pm, dovecot-stats-deliver and of course the Cacti template which may be messy to update.

Vlad Image  07/10/2013 12:13 :: Vlad

Thanks for the update! Most of the things work fine, besides the cacti template. Can I use the old cacti template with your new scripts?

Andreas Image  22/10/2013 14:59 :: Andreas

Hello!

It seems my post here two days ago is not show so I'll try it again. I am running dovecot 2.1.17. The only graph that is drawn with the Dovecot stats on Cacti tarball 20130920 is the session-graph. Where could it start to check if all the needed stuff is found and paths are right?

Andreas

Travis Image  23/10/2013 14:13 :: Travis

Glen,

I don't see a link to the cacti template in the article.  Am I missing something?

Travis

Glen Pitt-Pladdy Image  24/10/2013 18:55 :: Glen Pitt-Pladdy

Andreas - I am running 2.1.7 so the version is good. The data goes into the stats file specified (if you are using the example in the uloganalyser post then /var/local/snmp/mail). What is unique about the sessions is that they are picked up from doveadm, whereas everything else is from parsing log files which should be a big clue to what part is not working and where to investigate.

Glen Pitt-Pladdy Image  24/10/2013 18:58 :: Glen Pitt-Pladdy

Travis - the template is in the tarball along with all the plugin (you will still need uloganalyser from the other article), scripts and other bits needed to set this up: cacti_host_template_dovecot.xml

Travis Brown Image  27/10/2013 01:48 :: Travis Brown

Thanks Glen! I am not sure how I missed that before.

Mitch Image  06/11/2013 14:36 :: Mitch

Thanks for all your work on this!

I'm trying to import into the tarball 20131027 into Cacti 0.8.7i and I'm getting "Error: XML: Hash version does not exist". All the data, graph and host templates import but there is no name on any of them.

Am I missing something obvious?

Mitch Image  06/11/2013 14:52 :: Mitch

I have the same problem with the Cacti file for postfix also.

Glen Pitt-Pladdy Image  06/11/2013 18:55 :: Glen Pitt-Pladdy

This is very likely because the version these were done on is 0.8.8a and your version doesn't know about that. It's possible the ugly stuff in my "Cacti hack for forward compatibility" article will help (or really mess things up), otherwise I expect you will have to upgrade.

Alessio Image  30/04/2014 13:06 :: Alessio

Hi Glen,

I have found an issue with my dovecot log format (2.2.12):

dovecot.pm 20131027:419 /root/mail.log.0:191 unknown dovecot: Apr 29 23:59:59 pop01 dovecot: imap(pippo@pippo.com), session=<HPjxVzX4XgBdLtmW>: Disconnected: Logged out in=564 out=11430

dovecot.pm 20131027:419 /root/mail.log.0:194 unknown dovecot: Apr 29 23:59:59 pop01 dovecot: pop3(pippo@pippo.com), session=<98OajTX4eAC8Dqbj>: Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0, bytes=12/43

the difference from my log and your "regex" for log out is that my log starts with session=

I understand that this problem can be fixed by editing this line in dovecot.pm:

elsif ( $line =~ s/^(Disconnected)[:\s]*// or $line =~ s/^(proxy)\([^\)]+\): disconnecting [\da-f\.:]+ // )

but I'm not able, can you help me?

Thanks

Alessio Image  30/04/2014 21:03 :: Alessio

Hi Glen,

I solved the previous problem moving "session" at the end of the line but now your plugin is not any more able to count the in/out bytes.

Here a sample of my "Disconnected" log:

Apr 30 23:01:02 pop01eeh dovecot: imap(pippo@pippo.com): Disconnected: Logged out in=640 out=3883 session=<ACSb2Ej4iABtqHHz>
Apr 30 23:01:02 pop01eeh dovecot: pop3(pippo@pippo.com): Disconnected: Logged out top=0/0, retr=0/0, del=0/0, size=0, bytes=12/43, session=<jAaa2Ej4cAAuLOvd>

Can you help me?
Thanks

Glen Pitt-Pladdy Image  30/04/2014 21:17 :: Glen Pitt-Pladdy

What version of Dovecot and distro are you running? How have you changed the log format?

Alessio Image  05/05/2014 14:54 :: Alessio

Hi Glen,

I'm running Dovecot 2.2.12 build from source with a small change to logout format:

# Logout POP
pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o, session=<%{session}>

# Logout IMAP
imap_logout_format = in=%i out=%o session=<%{session}>

I added the "session" to logout for a law requirement (match the login and logout session).

Glen Pitt-Pladdy Image  06/05/2014 07:29 :: Glen Pitt-Pladdy

Hi, I's not practical for me to support customized log formats with this tool, however with some basic knowledge of Perl and regex you should be able to tweak the script to match your customization.

skeletor Image  16/12/2014 08:56 :: skeletor

Sub analyse at module dovecot.pm doesn't return any data. Sub wrapup works. What i am doing wrong?

Glen Pitt-Pladdy Image  22/12/2014 15:54 :: Glen Pitt-Pladdy

That's not really enough information to go on. Sub analyse is only called on new lines being found by uloganalyser, and then it filters the lines as per the regex. Currently working fine for me on Debian Wheezy, but if you are running something different or have changed log formats then you will need to debug and adapt the plugin for your needs.

voytek Image  12/02/2015 07:14 :: voytek

Glen,
thanks again for such great tools, all running well here.

if I want to upgrade my installation to the latest scripts you have here:

is it just sufficient to download latest tarball above, OR, do I also need to re-download templates, and, re import over 'old' ones ?

I'll just start with tarball..

Glen Pitt-Pladdy Image  12/02/2015 20:18 :: Glen Pitt-Pladdy

That depends a lot on where you are starting from. I've tried to keep the *-stats scripts as consistent as possible so newer ones just add more additional metrics. The uloganalyser plugins should have minimal impact between versions and at most just add more metrics for the *-stats scripts to pick up.

If Cacti templates have changed since the version you have been running then it's a much more tricky subject. Here weird stuff might happen when you import a newer templates. The safest way is to purge all old templates and graphs and start clean otherwise it's down to luck and manual clean-up afterwards if anything goes wrong. It's one area that I've seen some really messy stuff with Cacti.




Are you human? (reduces spam)
Note: Identity details will be stored in a cookie. Posts may not appear immediately