Glen Pitt-Pladdy :: BlogDebugging Cacti Problems | |||
Since I have quite a few Cacti templates I've created over the years, I get quite a few requests for help when they don't work. I think one of the big problems is that there's a lack of a "helicopter view" of how it all works, and what sort of things go wrong. This posting is an attempt to start changing that. Ingesting dataCacti has two basic ways of getting data in, and which to use depends on the type of data. I'm doing this in the reverse order since it starts off with least detail, and drills down, plus this is also the way that graphs are created in Cacti. BasicThis is where you have a fixed data set like Load Average. There is only one, or one set of data per device, so it's very simple. 3 - GraphGraphs are normally created from a template, for each device. The template will have associated Data Source(s) which it draws data from. Each data source is an RRD file and this is where the graphs are generated from. When a Graph is created from a template, Data Source(s) used by the graph are generated from their templates that are needed by the Graph. Note that removing the Graph doesn't remove the Data Sources unless you select to do that, and even then it doesn't remove the RRD files - they just stop being updated. 2 - Data SourceData sources are again normally created from a template, and then one or more may be used in a Graph. Data Sources can have one or more souces of data and each Data Source has an RRD file associated with it. Updating things in the Data Source settings doesn't updated the corresponding settings in the RRD file. You can either manually change them with rrdtool or delete the RRD file and it will be created again on the next sample with the new settings, but obviously you loose the data that was in it. Each Data Source has a Data Input Method associated with it which defines how to get the data. Along with that the settings to get the particular bit of data (eg. the OID for SNMP) 2 - Data Input MethodThis could be by SNMP or running a script that could collect local data or retrieve it remotely via some protocol (eg. ssh). My preference is to use SNMP since it's widely supported and provides the only viable "one size fits all" option. Scripts generally output white-space separated data that can then be parsed for each value. SNMP sticks with one data item per OID. IndexedWhen there could be multiple of something and you want to be able to discover and identify the things separately then this comes into play. It could be NICs, Drives, Mountpoints, or any number of things where each device could have a different number of configuration. In this case things are slightly different to accommodate this. 4 - GraphLike with the Basic case, Indexed Graphs have associated Data Source(s). Where things differ is that the Data Source(s). 3 - Data SourceSimilarly Data Sources define how to retrieve data and have an RRD file associated with them, but with a small difference. They are set to have an Indexed Data Input Method, and don't have specifics about retrieving the data. eg. there is no OID defined for SNMP Input Methods. The actual specifics come from a Data Query. 2 - Data QueryThis uses an XML file to define how to retrieve and group the data. This will provide a bunch of named sources and associate them with different Data Source names. For example a NIC might be grouped by device name (eg. eth0, eth1, eth2 ...) and then for each of those sources could be Packets In, Packets Out, Data In, Data Out, Discards etc., each mapped onto a corresponding Data Source. The XML file is where the details are like the OID pattern (not fixed) to use for SNMP and the like. The important thing to note is that when a Data Query is added to a device it will update showing the list of available groups / sub-devices:
Or on the New Graph Page:
An important thing to note here is the Green Rings on the right. These can be clicked to refresh the Index which could be useful if any of the devices have changed or there was a problem picking the data up. 1 - Data Input MethodLike with the Basic case, this provides a means to collect the data, be it a script, SNMP or whatever. The important thing is that this is formatted in a way that the Data Query can group the data as you need. Poller RunsWhat happens when the poller runs is it goes off and collects the necessary data using the methods configured and caches it. Then the cached data is committed to the RRD files (or on the first poll the RRD files are created). Once there is data in the RRD files Generally this is where problems start to be seen. DebuggingSince there are a number of defined steps that Cacti goes through we can check that thing are working at each stage. Device ConfigurationIt's not uncommon that devices are not picking up data from the start. This often means that for SNMP configured hosts that you won't see vaild info in the top of the Device page:
Checks:
LoggingWhen the Poller kicks off it will log in the Cacti log file assuming you haven't disabled this. On Debian based systems this is /var/log/cacti/cacti.log and will have something like: 1/08/2015 08:25:15 PM - CMDPHP: Poller[0] Host[1] DS[2366] WARNING: Result from SNMP not valid. Partial Result: U So here right away you will see some interesting information. In this case we get a load of stats, but we also get problems with some Data Sources and the associated numbers and in one case the SNMP OID are displayed. In cases where we have "U" this is a deliberate thing in the collection scripts to signify invalid data. It could be that for example a particular drive type doesn't have some SMART parameters being polled. Now, one thing you can do is actually try and manually collect data. If it's a script then try running the script command used to retrieve data. If it's SNMP then run some snmpwalk commands up the appropriate OIDs for the data as in my SNMP Basics article. Increased LoggingLogging can be increased in the settings:
Start increasing the Log Level and check what you get. DEBUG is likely the highest you will need to go, but you could push it further if you need. 11/08/2015 08:35:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/####_lmsensor_temp5_2667.rrd --template lmsensor_temp5 1447014901:33.0 Now you are actually getting details like the actual OID being polled and the values returned. This allows you to validate the particular data that is being returned and the files it's going into, and the command being run to put the data into the RRD file. RRD FileWe should really check the RRD file. To do this run: $ rrdtool info somefile.rrd Key things to look out for are last_ds which shows the last value and last_update which is the time as Unix Epoch when the update was made. You can convert the time at Epoch Converter. Another thing to look at is the min and max since if the data being added falls outside of those then the data will become a NaN (Not a Number) value. One thing I've seen recently is a case where the RRD files ended up created with invalid step and minimal_heartbeat settings. These relate to the polling rate that Cacti is set to which in most cases will be 300 seconds (5 minutes). In this case step will be set to 300, and typically minimal_heartbeat is set to double this. If polling occurs at a longer interval than minimal_heartbeat then data will not be considered valid in the RRD file (NaN). You can also use the "dump" function to examine all the data in the file and see if it's being added consistently. If data is not being added then you may want to check permissions of the file, directory etc. to verify if it's all right. If it is then try a manual update based on the data in the log. You will want to add time to Epoch for the sample to match the next data sample for the file. Remember to also "su - CactiUser" to ensure that you are adding data exactly the same way as it would normally. One small difference to the command though is to use "updatev" instead of "update" which essentially means Verbose: $ rrdtool updatev somefile.rrd --template smart_10 Epoch:90 Try adding more than one sample, incrementing the Epoch appropriately each time. If you have a sample out of range (may require more than one) you could get something like: $ rrdtool updatev somefile.rrd --template smart_10 Epoch:999 GraphsOnce you have valid data in your RRD file, and remember it may take more than one and there are problems with graphs then you can go and find the individual graph that is giving problems and click on "*Turn Off Graph Debug Mode" which will then print a load of additional data on the command being used to generate the graph and any errors it encounters:
Graph But No Data ShownWith Indexed sources a problem I've seen is that the graphs get out of sync with the device (eg. a NIC was added, or devices where detected in different order due to a reboot). In this case clicking the Green Rings with the Data Query (see above) should do the trick. Another check to make on Data Queries is that all the Data Sources are correctly associated. This can be found in Console->Data Queries->select query (at this stage check for "Successfully located XML file" in Green)->each template:
One thing to check is that the checkbox on the right is checked. Changing Data Source SettingsA common problem I see with Cacti is that when a problem is found, changes are made to correct this but nothing appears to happen. This is often down to the fact that Cacti (at least the versions I've seen) does not change already-existing RRD files. The RRD files are created on the first polling cycle from the settings in the Data Source, but after that there are no changes made to the RRD configuration. I've seen plugins for manipulating RRD files, but in many cases the most practical things to do would be to use "rrdtool tune" to manually change RRD file settings where possible, or in cases where the Data Source has never worked then simply deleting the RRD file and letting Cacti recreate it with the correct settings on the next polling cycle is probably the most practical thing to do. Other StuffOne thing that I've noticed is that some errors are due to timeouts (eg. SNMP) which can result in part of the Data Sources for a device not updating and other updating, while a small group may be intermittent. In this case you will need to tune the timeouts and number of poller threads to avoid this. |
|||
|
This is a bunch of random thoughts, ideas and other nonsense, and is not intended to be taken seriously. I'm experimenting and mostly have no idea what I am doing with most of this so it should be taken with cuation and at your own risk. Intrustive technologies are minimised where possible. For the purposes of reducing abuse and other risks hCaptcha is used and has it's own policies linked from the widget.
Copyright Glen Pitt-Pladdy 2008-2023
|
|||
