Since the below is a little long, I put my question at the top: how do I
troubleshoot awful exported resources performance in puppet and is there
anything I can tweak to get it to run under 10 minutes in larger
I have a fairly modest environment (118 nodes, but prod will be at least
twice as large). I'm trying to move my distributed nagios setup to one
based on exported resources and it works brilliantly in a smaller env (35
nodes), but with 118 nodes it's not so great anymore.
All the hosts export a nagios host and services (up to 20) with
corresponding commands, plus files for them. It's all based on this:
https://github.com/camptocamp/puppet-nagios , but fixed so that it's
working :). I appreciate it's not the most efficient way of doing things,
but it makes my setup insanely easy to configure, comparing to the previous
one, with a combination of static files, templates, multiple 'profiles',
duplicated configs, etc.
In the small env puppet agent takes about 2 minutes to run on nagios server
and collect resources.
In the larger env it takes about 70 minutes, if it manages to finish at
all. Initially, as a "quick" test, I was running puppetdb without postgres
and had to give it 5GB to get it to finish at all (70 mins). With postgres
8.4, load on the puppetmaster is significantly reduced, but with 512MB for
puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb
still runs out of memory. I set it to 1GB and puppedb just crashed again
(I've got dumps). Trying with 2GB now. I haven't fiddled with thread
settings, but my puppet agents aren't deamonized or 'croned', I run them
using mcollective or manually. So there's only a single puppet agent
running during this test, on the core nagios server. It seems that there's
a ruby process taking 100% of one core during this run and nothing else
"dramatic" seems to be happening (except for puppetdb dying of course).
It's all on centos 5.8 with puppetlabs ruby and puppet packages, on
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/bLnpN2PTnQsJ.
To post to this group, send email to email@example.com.
To unsubscribe from this group, send email to firstname.lastname@example.org.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.