FAQ
Hi all,

For months now I've been using Puppet with PuppetDB backend to manage my
Nagios configs. We now have 1200+ services being checked and including
servicedependency, host, hostdependency, command and other Nagios
resources being managed, that's about 3000 resources in total.
Unfortunately this means the monitoring server takes about 8 minutes to
perform a Puppet run.

I'm not exactly sure what the bottleneck is - whether it's doing 3000
calls to PuppetDB (although the load on the PuppetDB server and its
Postgres database are low), or whether it is parsing nagios_service.cfg
that causes the slowdown. It's about 500kb and 11,000 lines.

I know it is possible to squirt the Nagios config into several smaller
files when the resources are collected on the monitoring server, but I
gather this also means that Nagios resources cannot be purged if the
default file locations are not used. The ability to purge old configs
automatically is essential to our environment.

So does anyone have any suggestions on how to improve the performance of
Puppet when handling large Nagios configs?

Cheers,
Jonathan

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Ken Barber at May 24, 2013 at 2:59 pm
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.

    The most performant work-around I've seen for this is that you can use
    the nagios_* resources to export, but then use custom collectors that
    query PuppetDB (using Dalen's puppetdbquery for example:
    https://github.com/dalen/puppet-puppetdbquery) that generates custom
    templates for your nagios_*.cfg files. If nagios_service.cfg is the
    main bottleneck, you can focus on this one to start with.

    This way you continue to use the model of the nagios_* resources, but
    at collection time you improve performance.

    ken.

    On Fri, May 24, 2013 at 1:53 PM, Jonathan Gazeley
    wrote:
    Hi all,

    For months now I've been using Puppet with PuppetDB backend to manage my
    Nagios configs. We now have 1200+ services being checked and including
    servicedependency, host, hostdependency, command and other Nagios resources
    being managed, that's about 3000 resources in total. Unfortunately this
    means the monitoring server takes about 8 minutes to perform a Puppet run.

    I'm not exactly sure what the bottleneck is - whether it's doing 3000 calls
    to PuppetDB (although the load on the PuppetDB server and its Postgres
    database are low), or whether it is parsing nagios_service.cfg that causes
    the slowdown. It's about 500kb and 11,000 lines.

    I know it is possible to squirt the Nagios config into several smaller files
    when the resources are collected on the monitoring server, but I gather this
    also means that Nagios resources cannot be purged if the default file
    locations are not used. The ability to purge old configs automatically is
    essential to our environment.

    So does anyone have any suggestions on how to improve the performance of
    Puppet when handling large Nagios configs?

    Cheers,
    Jonathan

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Alexander Bien at May 25, 2013 at 8:11 am

    On 24.05.2013 16:58, Ken Barber wrote:
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.
    I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
    nagios_* files and >50k lines of config. The puppet run takes aprox 45
    minutes. I have heard people on irc say that its not uncommon.

       - alex

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • David Schmitt at May 25, 2013 at 4:24 pm

    On 2013-05-25 10:11, Alexander Bien wrote:
    On 24.05.2013 16:58, Ken Barber wrote:
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.
    I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
    nagios_* files and >50k lines of config. The puppet run takes aprox 45
    minutes. I have heard people on irc say that its not uncommon.
    You can cut quite a bit off that time by using templates and files (one
    per service/host) and purging on the base directory.

    Regards, David


    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jonathan Gazeley at Jun 5, 2013 at 1:53 pm

    On 25/05/13 17:24, David Schmitt wrote:
    On 2013-05-25 10:11, Alexander Bien wrote:
    On 24.05.2013 16:58, Ken Barber wrote:
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.
    I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
    nagios_* files and >50k lines of config. The puppet run takes aprox 45
    minutes. I have heard people on irc say that its not uncommon.
    You can cut quite a bit off that time by using templates and files
    (one per service/host) and purging on the base directory.
    Hi David,

    Sorry to reply to a stale thread. Please could you explain a bit more on
    your solution? Presumably I need to pas some extra arguments to my
    spaceship operator that collects the Nagios resources, but how is the
    directory-level purging done?

    Thanks,
    Jonathan

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • David Schmitt at Jun 5, 2013 at 7:57 pm

    On 2013-06-05 15:53, Jonathan Gazeley wrote:
    On 25/05/13 17:24, David Schmitt wrote:
    On 2013-05-25 10:11, Alexander Bien wrote:
    On 24.05.2013 16:58, Ken Barber wrote:
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.
    I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
    nagios_* files and >50k lines of config. The puppet run takes aprox 45
    minutes. I have heard people on irc say that its not uncommon.
    You can cut quite a bit off that time by using templates and files
    (one per service/host) and purging on the base directory.
    Hi David,

    Sorry to reply to a stale thread. Please could you explain a bit more on
    your solution? Presumably I need to pas some extra arguments to my
    spaceship operator that collects the Nagios resources, but how is the
    directory-level purging done?
    I'm doing it this way:

    file {
      "/etc/nagios/conf.d":
       ensure => directory,
       source => "puppet:///modules/nagios/empty",
       ignore => [ '.gitkeep' ],
       recurse => true,
       force => true,
       purge => true;
    }

    File<<| tag == 'nagios::service' |>>

    define nagios::service(...) {
      @@file {
       "/etc/nagios/conf.d/${host}_service_${name}":
        content => template("nagios/service.erb"),
        tag => 'nagios::service',
        ...;
    }

    This way you collect a minimal number of simple resources. This still is
    dog slow when you have a couple of thousand services, but it sure beats
    the heck out of parsing the complete nagios config for each nagios_*
    resource.



    Regards, David

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Chris Handy at Nov 11, 2013 at 9:29 pm
    If you are looking to implement it via the puppetdbquery module that Ken
    was referring to earlier, here is some code to do so with. I recently
    implemented this approach with my own setup and it worked out great. About
    2500 nagios_service resources went from taking 20-30 minutes to about 2.
      Only issue i ran into was with puppet dashboard when the size of the
    report log became too big to save for one run. I had to alter the mysql DB
    columns from text to mediumtext and now its all good.

    https://gist.github.com/richardc/4714701
    On Wednesday, June 5, 2013 3:57:22 PM UTC-4, David Schmitt wrote:
    On 2013-06-05 15:53, Jonathan Gazeley wrote:
    On 25/05/13 17:24, David Schmitt wrote:
    On 2013-05-25 10:11, Alexander Bien wrote:
    On 24.05.2013 16:58, Ken Barber wrote:
    Its probably the nagios resource, its well known for not scaling when
    the files contain a lot of entries. That would explain why your
    PuppetDB is showing low load - as its probably not the bottleneck.
    I second that. We have mysql storeconfigs, about 5000 Services, 2,5 MB
    nagios_* files and >50k lines of config. The puppet run takes aprox 45
    minutes. I have heard people on irc say that its not uncommon.
    You can cut quite a bit off that time by using templates and files
    (one per service/host) and purging on the base directory.
    Hi David,

    Sorry to reply to a stale thread. Please could you explain a bit more on
    your solution? Presumably I need to pas some extra arguments to my
    spaceship operator that collects the Nagios resources, but how is the
    directory-level purging done?
    I'm doing it this way:

    file {
    "/etc/nagios/conf.d":
    ensure => directory,
    source => "puppet:///modules/nagios/empty",
    ignore => [ '.gitkeep' ],
    recurse => true,
    force => true,
    purge => true;
    }

    File<<| tag == 'nagios::service' |>>

    define nagios::service(...) {
    @@file {
    "/etc/nagios/conf.d/${host}_service_${name}":
    content => template("nagios/service.erb"),
    tag => 'nagios::service',
    ...;
    }

    This way you collect a minimal number of simple resources. This still is
    dog slow when you have a couple of thousand services, but it sure beats
    the heck out of parsing the complete nagios config for each nagios_*
    resource.



    Regards, David
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/2858c151-1647-4bcd-bd3d-6071451aedf1%40googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedMay 24, '13 at 12:53p
activeNov 11, '13 at 9:29p
posts7
users5
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase