[munin-users] high peak in network graph - how to prevent it?

First consider this:
http://munin.projects.linpro.no/wiki/HowToWritePlugins#DERIVEvs.COUNTER

But it sounds like you're duplicating this:
http://munin.projects.linpro.no/browser/trunk/plugins/node.d/snmp__if_.in
(included like that in 1.3 series and soon to come 1.4.0), so looking at
it may help. I guess the .max is needed to avoid positive spikes.

Nicolai

[...]
[...]
Thanks for the pointers! It really helped!


Daniel

Nicolai wrote excellent suggestions, but I have another one.

There is a workaround, which I use to remove unneeded peaks from various
rrd data, which happens to me occasionaly from time to time after
various restarts, for example interface counters/traffic after server
restart. Basicaly I extract rrd data to xml and then fix the peak using
perl (you may need other value to be fixed than in my script). I think I
found similar script somewhere over Internet and adapted it to my
purpose. Remember to backup the *.rrd files before you do that. ;-)

#!/bin/sh

for i in `ls *.rrd`; do
mv $i ${i}.bak
rrdtool dump ${i}.bak | perl -pe 's/(.*) \d\.\d+e\+1\d
(<\/v>.*)/$1 0.0000000000e+00 $2/' > ${i}.xml
rrdtool restore ${i}.xml ${i}
done

2009/11/24 Maciej Wierzbicki <[email protected]>
Actually, you don't even need to parse the xml data with Perl. Just modify
directly the rrds files with rrdtune and dump/restore the data. rrdrestore
will wipe out the erroneous data. Something like :

rrdtool tune ${RRD_FILE} --maximum 42:${MAXIMUM_VALUE}
(rrdtool dump ${RRD_FILE} | rrdtool restore --range-check - ${RRD_FILE}.new)
&& \
mv ${RRD_FILE} ${RRD_FILE}.bak && \
mv ${RRD_FILE}.new ${RRD_FILE}
# Optionally remove the maximum
rrdtool tune ${RRD_FILE} --maximum 42:U

Wow, Maciej and Steve, great ideas, thanks a lot!


Daniel

[...]

WRT to Nicolai's suggestion:

My ${name}.min configuration option doesn't seem to work :\

My script produces this for config == $1:
echo 'graph_order in out'
echo "graph_title eth0 traffic (ifHC)"
echo "graph_args --base 1000 --lower-limit 0 --units-exponent 6 \
--upper-limit 1000000000 --rigid"
echo 'graph_vlabel bits in/out per ${graph_period}'
echo 'graph_category network'
echo 'in.label in'
echo 'in.type DERIVE'
echo 'in.min 0'
echo 'in.cdef in,8,*'
echo 'in.draw AREA'
echo 'out.label out'
echo 'out.type DERIVE'
echo 'out.min 0'
echo 'out.cdef out,8,*'
echo 'out.draw LINE1'

Is there any catch regarding the use of the .min value? When I restart snmpd
now, I get -20,-30Gb values on the graphs, despite of the .min option. This
is a way smaller value than before with the COUNTER type, I admit, but still,
it would be nice eliminate this negative value :)

Thanks in advance,

Daniel

You may be bumping into a problem that has been reported once
previously: somehow some rrd files does not get re-tuned properly when
the min/max settings are changed.

Can you please run "rrdtool tune" on all of the rrd files created by the
plugin and see if the min settings are according to the plugin?

If they're not there was a posting earlier today about retuning and
"normalizing" the rrd file by export/import.

I'd also like to know if they're not.

Nicolai

[...]
[...]
[...]
Those advices worked like charm, now I'm not just protected from future peaks,
but the past peaks got removed from the graphs too.

Thanks again for the great support!


Daniel

Bummer. What version of munin are you running?
It would be very appreciated if someone entered this, including how to
fix the rrd files, in the FAQ on the wiki.

Thanks,
Nicolai

2009/11/26 Nicolai Langfeldt <[email protected]>
Just did it.
http://munin-monitoring.org/wiki/faq#Q:Ieditedmyplugintohaveminmaxvaluesbuttheyarenottakeninaccount

Strange i have the ACL to submit changes but not preview them :-)

thanks a lot :-)

Nicolai

munin 1.2.6 from Debian lenny. Maybe there will be munin-1.4 package after its
release on backports in the near future.
Thanks!


Daniel

Btw, I just discovered an old page in the wiki almost about this.
http://munin.projects.linpro.no/wiki/SpikeRemoval