Combined collectd stats for iptables

With collectd, system health and application performance data can be collected, transmitted, and stored. By choosing from the many provided plugins, commonly used metrics can easily be aggregated into rrd databases. Various other projects then provide the functionality to postprocess these for aggregation and graph visualizations.

With its iptables plugin, firewall rule counters can be tracked conveniently. However, this is limited to a single counter instance per rule, without the possibility to combine several rules that serve a common purpose. The patch provided below fixes this by automatically combining counter values for rules of the same type.

Statistics for iptables with collectd

The iptables plugin can be enabled by stating the table and chain to be watched. The following collectd.conf allows to track byte and packet counters for all rules that have an arbitrary comment set:

LoadPlugin iptables
<Plugin iptables>
    Chain "filter" "INPUT"
</Plugin>

However, this assumes that each rule comment is unique. As multiple rules could serve the same purpose, it might be convenient to combine rules that share the same comment. In addition, a single rule might get implicitly splitted when certain multi-matches are used, for example:

$ iptables -A INPUT -d 10.0.0.0/8,224.0.0.0/4,240.0.0.0/4 -m comment --comment "duplicate comment" -j DROP
$ iptables -L -n -v -x | grep "duplicate comment"
0  0  DROP  all  --  0.0.0.0/0  10.0.0.0/8   /* duplicate comment */
0  0  DROP  all  --  0.0.0.0/0  224.0.0.0/4  /* duplicate comment */
0  0  DROP  all  --  0.0.0.0/0  240.0.0.0/4  /* duplicate comment */

As for this case, comments that are not unique will cause collectd to insert samples for the same counter multiple times at once. An indication for this happening are iptables-related log entries involving duplicate timestamps. In especially the minimum one second step error suggests repeated, concurrent insertions for the same counter:

cu_rrd_create_file: File ... is already being created.
uc_update: Value too old: name = ...; value time = ...; last cache update = ...;
rrd_update_r (...) failed: ...: illegal attempt to update using time ... when last update time is ... (minimum one second step)

As also proposed in collectd bug #493, a possible solution is to sum up the counters of rules that share the same comment beforehand.

Allow duplicate rule comments per chain

Instead of dispatching packet and byte counters for each commented rule directly, their values are buffered in memory first. Statistics of rules that share the same comment (and of course table and chain) are summed up on-the-fly. Afterwards, the coalesced counters are handed over in bulk for further internal processing.

Please note that adding up counters before passing them to the rrd subsystem might cause overflow issues. Usually, counter wraps or resets are detected and handled accordingly. While a 64bit overflow of the total should thus not be an issue, resets or wraps of one counter in the sum could go undetected – causing a single, faulty sampling point in rare cases. So please re-check applicability for scenarios that utterly require reliable values under high traffic over a long runtime.

Module patch and installation

For building the new module, first download and extract the patch from below and the original sources from github. I based it on the versions 5.5.1 and 5.7.2 that some of my distributions use, but others may work, too. After applying the patch and compiling everything including the changed module, the resulting plugin is ready to use.

# download/extract vanilla sources and patch
cd collectd-5.7.2
patch -p1 < ../iptables-5.7.2.c.patch
./configure --enable-iptables
make
sudo install -b -s -v -t /usr/lib/collectd/ src/.libs/iptables.so
sudo systemctl restart collectd # in case of systemd
# might want to clean affected databases in /var/lib/collectd/rrd

Building the plugin requires the libiptc and iptables-dev libraries and headers, which should be easily installable on common distributions. collectd itself has only very basic dependencies.