FAQ
Hi,

Since the below is a little long, I put my question at the top: how do I
troubleshoot awful exported resources performance in puppet and is there
anything I can tweak to get it to run under 10 minutes in larger
environments?

I have a fairly modest environment (118 nodes, but prod will be at least
twice as large). I'm trying to move my distributed nagios setup to one
based on exported resources and it works brilliantly in a smaller env (35
nodes), but with 118 nodes it's not so great anymore.

All the hosts export a nagios host and services (up to 20) with
corresponding commands, plus files for them. It's all based on this:
https://github.com/camptocamp/puppet-nagios , but fixed so that it's
working :). I appreciate it's not the most efficient way of doing things,
but it makes my setup insanely easy to configure, comparing to the previous
one, with a combination of static files, templates, multiple 'profiles',
duplicated configs, etc.

In the small env puppet agent takes about 2 minutes to run on nagios server
and collect resources.

In the larger env it takes about 70 minutes, if it manages to finish at
all. Initially, as a "quick" test, I was running puppetdb without postgres
and had to give it 5GB to get it to finish at all (70 mins). With postgres
8.4, load on the puppetmaster is significantly reduced, but with 512MB for
puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb
still runs out of memory. I set it to 1GB and puppedb just crashed again
(I've got dumps). Trying with 2GB now. I haven't fiddled with thread
settings, but my puppet agents aren't deamonized or 'croned', I run them
using mcollective or manually. So there's only a single puppet agent
running during this test, on the core nagios server. It seems that there's
a ruby process taking 100% of one core during this run and nothing else
"dramatic" seems to be happening (except for puppetdb dying of course).

It's all on centos 5.8 with puppetlabs ruby and puppet packages, on
apache+passenger:

ruby-1.8.7.370-1.el5
ruby-rdoc-1.8.7.370-1.el5
rubygem-stomp-1.2.2-1.el5
rubygem-json-1.4.6-2.el5
rubygem-fastthread-1.0.7-1
rubygem-rake-0.8.7-2
rubygem-daemon_controller-0.2.5-1
rubygem-passenger-native-3.0.12-1
puppetdb-terminus-1.0.5-1.el5
puppet-server-3.0.2-1.el5
ruby-libs-1.8.7.370-1.el5
ruby-irb-1.8.7.370-1.el5
rubygems-1.3.7-1.el5
ruby-shadow-1.4.1-7
libselinux-ruby-1.33.4-5.7.el5
ruby-augeas-0.4.1-1
puppet-3.0.2-1.el5
rubygem-rack-1.1.0-2.el5
rubygem-passenger-3.0.12-1
rubygem-passenger-native-libs-3.0.12-1_1.8.7.370
puppetdb-1.0.5-1.el5

Passenger config:

PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
PassengerMaxRequests 1000
PassengerStatThrottleRate 120


Regards,
Daniel

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/bLnpN2PTnQsJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • David Schmitt at Jan 21, 2013 at 4:53 pm

    On 21.01.2013 14:05, Daniel wrote:
    Hi,

    Since the below is a little long, I put my question at the top: how do I
    troubleshoot awful exported resources performance in puppet and is there
    anything I can tweak to get it to run under 10 minutes in larger
    environments?
    There is a "simple" answer: puppetdb.

    There'll still be bottlenecks to wring, but collection performance won't
    be one of them.


    Best Regards, David

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • GRANIER Bernard (MORPHO) at Jan 21, 2013 at 4:58 pm
    Hi,

    Is there a way to define a variable in puppet.conf file ?

    Something like :
    My_var = my_value in puppet.conf

    Then to be able to use it in manifest with something like :
    file {custo.txt':
    path=> $my_value,
    ensure=> present,
    mode=> 0640,
    content=> "this content",
    }

    Sincerly,

    Bernard Granier

    #
    " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
    #

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Brian Mathis at Jan 21, 2013 at 5:09 pm
    Something like this would typically go in your manifest files, not the
    puppet.conf file. I think you will find the Puppet language guide
    instructive, specifically the page on variables:
    http://docs.puppetlabs.com/puppet/2.7/reference/lang_variables.html

    You should think of Puppet as more of a development language than a
    tool that you simply configure using a .conf file. The site.pp is
    simply the entry point into the structure you build with the
    manifests, and you can do quite a lot with the language.


    ❧ Brian Mathis


    On Mon, Jan 21, 2013 at 11:58 AM, GRANIER Bernard (MORPHO)
    wrote:
    Hi,

    Is there a way to define a variable in puppet.conf file ?

    Something like :
    My_var = my_value in puppet.conf

    Then to be able to use it in manifest with something like :
    file {custo.txt':
    path=> $my_value,
    ensure=> present,
    mode=> 0640,
    content=> "this content",
    }

    Sincerly,

    Bernard Granier

    #
    " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
    #

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • GRANIER Bernard (MORPHO) at Jan 21, 2013 at 5:17 pm
    Thanks for the advice, I will use variables to define specific values,

    Sincerely,


    Bernard Granier


    -----Original Message-----
    From: puppet-users@googlegroups.com On Behalf Of Brian Mathis
    Sent: Monday, January 21, 2013 6:09 PM
    To: puppet-users@googlegroups.com
    Subject: Re: [Puppet Users] puppet configuration and variables ?

    Something like this would typically go in your manifest files, not the puppet.conf file. I think you will find the Puppet language guide instructive, specifically the page on variables:
    http://docs.puppetlabs.com/puppet/2.7/reference/lang_variables.html

    You should think of Puppet as more of a development language than a tool that you simply configure using a .conf file. The site.pp is simply the entry point into the structure you build with the manifests, and you can do quite a lot with the language.


    ❧ Brian Mathis

    On Mon, Jan 21, 2013 at 11:58 AM, GRANIER Bernard (MORPHO) wrote:
    Hi,

    Is there a way to define a variable in puppet.conf file ?

    Something like :
    My_var = my_value in puppet.conf

    Then to be able to use it in manifest with something like :
    file {custo.txt':
    path=> $my_value,
    ensure=> present,
    mode=> 0640,
    content=> "this content",
    }

    Sincerly,

    Bernard Granier

    #
    " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
    #

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

    #
    " This e-mail and any attached documents may contain confidential or proprietary information. If you are not the intended recipient, you are notified that any dissemination, copying of this e-mail and any attachments thereto or use of their contents by any means whatsoever is strictly prohibited. If you have received this e-mail in error, please advise the sender immediately and delete this e-mail and all attached documents from your computer system."
    #

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Luke Bigum at Jan 21, 2013 at 6:10 pm
    Hi Daniel,
    On Monday, January 21, 2013 1:05:26 PM UTC, Daniel wrote:

    In the larger env it takes about 70 minutes, if it manages to finish at
    all. Initially, as a "quick" test, I was running puppetdb without postgres
    and had to give it 5GB to get it to finish at all (70 mins). With postgres
    8.4, load on the puppetmaster is significantly reduced, but with 512MB for
    puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb
    still runs out of memory. I set it to 1GB and puppedb just crashed again
    (I've got dumps). Trying with 2GB now. I haven't fiddled with thread
    settings, but my puppet agents aren't deamonized or 'croned', I run them
    using mcollective or manually. So there's only a single puppet agent
    running during this test, on the core nagios server. It seems that there's
    a ruby process taking 100% of one core during this run and nothing else
    "dramatic" seems to be happening (except for puppetdb dying of course).
    Given enough RAM it doesn't sound like PuppetDB is the problem any more, is
    that correct?

    The Ruby process is most likely a Puppet Master thread doing the catalog
    construction. I think your suffering from a similar problem that we had
    recently, where it's not specifically resource collection that's taking up
    all the time, it's the Puppet Master turning the exported resources
    information into one enormous catalog that takes too long.

    We got around this by bypassing exported resources and querying the
    information from PuppetDB directly and using that information in a
    template. I suggested the following to another user a few days ago in this
    thread:

    https://groups.google.com/forum/#!topic/puppet-users/X6Lm-0_etbA

    -Luke

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/GOV7Nh_co1EJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Daniel at Jan 22, 2013 at 1:56 pm

    On Monday, January 21, 2013 1:05:26 PM UTC, Daniel wrote:
    In the larger env it takes about 70 minutes, if it manages to finish at
    all. Initially, as a "quick" test, I was running puppetdb without postgres
    and had to give it 5GB to get it to finish at all (70 mins). With postgres
    8.4, load on the puppetmaster is significantly reduced, but with 512MB for
    puppetdb (128 + 1MB per node, and then double it for good measure) puppetdb
    still runs out of memory. I set it to 1GB and puppedb just crashed again
    (I've got dumps). Trying with 2GB now. I haven't fiddled with thread
    settings, but my puppet agents aren't deamonized or 'croned', I run them
    using mcollective or manually. So there's only a single puppet agent
    running during this test, on the core nagios server. It seems that there's
    a ruby process taking 100% of one core during this run and nothing else
    "dramatic" seems to be happening (except for puppetdb dying of course).
    Given enough RAM it doesn't sound like PuppetDB is the problem any more,
    is that correct?

    The Ruby process is most likely a Puppet Master thread doing the catalog
    construction. I think your suffering from a similar problem that we had
    recently, where it's not specifically resource collection that's taking up
    all the time, it's the Puppet Master turning the exported resources
    information into one enormous catalog that takes too long.

    We got around this by bypassing exported resources and querying the
    information from PuppetDB directly and using that information in a
    template. I suggested the following to another user a few days ago in this
    thread:

    https://groups.google.com/forum/#!topic/puppet-users/X6Lm-0_etbA
    Hi Luke,

    This sounds like a sensible workaround, I will definitely have a look. I
    haven't yet had enough time to look at the issue properly, but it seems
    that this very long time is indeed consumed by catalog construction.
    Puppetdb fails after this is finished, so it seems that it dies when nagios
    host tries to report its catalog back.

    To be honest it's very disappointing that puppet and puppetdb can't handle
    few thousand exported resources and that workarounds like that are needed.
    I might need to look at alternatives.


    Regards,
    Daniel

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/uXWPXZLe_FgJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ken Barber at Jan 22, 2013 at 3:05 pm

    This sounds like a sensible workaround, I will definitely have a look. I
    haven't yet had enough time to look at the issue properly, but it seems that
    this very long time is indeed consumed by catalog construction. Puppetdb
    fails after this is finished, so it seems that it dies when nagios host
    tries to report its catalog back.
    Do you mean it dies from an OOM when it tries to report the catalogue back?

    ken.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Daniel Siechniewicz at Jan 22, 2013 at 3:19 pm

    On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber wrote:
    This sounds like a sensible workaround, I will definitely have a look. I
    haven't yet had enough time to look at the issue properly, but it seems that
    this very long time is indeed consumed by catalog construction. Puppetdb
    fails after this is finished, so it seems that it dies when nagios host
    tries to report its catalog back.
    Do you mean it dies from an OOM when it tries to report the catalogue back?
    Yes, that's what it looks like. Of course I can prevent it by giving
    it more memory (which I did), but I already have postgres backed
    puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
    single host (OK, with thousands of exported resources to collect and
    process) that takes about 70 minutes can still kill it. This waiting
    70 minutes for it to die is an insult to injury... Overall not great.
    I'm happy to redo this setup if I'm doing something wrong, but it just
    seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
    nodes, 70 minutes).

    Regards,
    Daniel

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ken Barber at Jan 22, 2013 at 4:16 pm
    How many resources are we talking for the 100 nodes?

    If you're running puppetdb one way to do this is to get a report on
    the nagios server that does the collection:

    curl -G -H 'Accept: application/json' 'http://puppetdb:8080/resources'
    --data-urlencode 'query=["=", ["node", "name"], "nagiosserver"]' | p
    resource | wc -l

    (this at least works on Puppetdb 1.0.5 - as JSON output is pretty-printed)

    It would be interesting to see if this is an artifact of collection or
    the nagios resource specifically, although the only good way I know to
    do this is to perhaps replace the nagios exported resource cases with
    something simple - such as a noop style resource like 'whit' or
    'anchor'.

    When you run puppet agent -t ... if you do a summarize what is the
    result? You should get something like this:

    # puppet agent -t --summarize
    info: Caching catalog for puppetdb1.vm
    info: Applying configuration version '1358813527'
    notice: exported
    notice: /Stage[main]//Node[puppetdb1.vm]/Notify[exported]/message:
    defined 'message' as 'exported'
    notice: asdfasdfsasdf
    notice: /Stage[main]//Node[puppetdb1.vm]/Notify[foo]/message: defined
    'message' as 'asdfasdfsasdf'
    notice: Finished catalog run in 0.03 seconds
    Changes:
    Total: 2
    Events:
    Total: 2
    Success: 2
    Resources:
    Changed: 2
    Out of sync: 2
    Skipped: 6
    Total: 9
    Time:
    Filebucket: 0.00
    Notify: 0.00
    Config retrieval: 0.90
    Total: 0.90
    Last run: 1358813994
    Version:
    Config: 1358813527
    Puppet: 2.7.18

    I think Luke's suggestion about tapping the information in puppetdb
    using functions is not a bad work-around however, but its
    disappointing that exported resources in this case isn't just
    _working_ :-(.

    ken.

    On Tue, Jan 22, 2013 at 3:19 PM, Daniel Siechniewicz
    wrote:
    On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber wrote:
    This sounds like a sensible workaround, I will definitely have a look. I
    haven't yet had enough time to look at the issue properly, but it seems that
    this very long time is indeed consumed by catalog construction. Puppetdb
    fails after this is finished, so it seems that it dies when nagios host
    tries to report its catalog back.
    Do you mean it dies from an OOM when it tries to report the catalogue back?
    Yes, that's what it looks like. Of course I can prevent it by giving
    it more memory (which I did), but I already have postgres backed
    puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
    single host (OK, with thousands of exported resources to collect and
    process) that takes about 70 minutes can still kill it. This waiting
    70 minutes for it to die is an insult to injury... Overall not great.
    I'm happy to redo this setup if I'm doing something wrong, but it just
    seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
    nodes, 70 minutes).

    Regards,
    Daniel

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Daniel at Jan 23, 2013 at 10:28 pm
    This is now reported here:

    http://projects.puppetlabs.com/issues/18804

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/ZpyFiFkYjawJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jcbollinger at Jan 22, 2013 at 5:26 pm

    On Tuesday, January 22, 2013 9:19:02 AM UTC-6, Daniel wrote:
    On Tue, Jan 22, 2013 at 3:04 PM, Ken Barber wrote:
    This sounds like a sensible workaround, I will definitely have a look.
    I
    haven't yet had enough time to look at the issue properly, but it seems
    that
    this very long time is indeed consumed by catalog construction.
    Puppetdb
    fails after this is finished, so it seems that it dies when nagios host
    tries to report its catalog back.
    Do you mean it dies from an OOM when it tries to report the catalogue
    back?

    Yes, that's what it looks like. Of course I can prevent it by giving
    it more memory (which I did), but I already have postgres backed
    puppetdb and had to give puppetdb 3GB, or a puppet agent run on a
    single host (OK, with thousands of exported resources to collect and
    process) that takes about 70 minutes can still kill it. This waiting
    70 minutes for it to die is an insult to injury... Overall not great.
    I'm happy to redo this setup if I'm doing something wrong, but it just
    seems like this is exponential (30-odd nodes - 2 minutes, 100-odd
    nodes, 70 minutes).
    This sounds like a memory-management problem. If you're running up against
    (or close to) a memory limit, then the Java-based puppetdb will start doing
    a lot of garbage collection as available heap memory becomes scarce. That
    will greatly slow down the whole Java VM, especially so if the GC runs
    don't free much memory. Also, if you give the VM more memory than
    available RAM then you will have slowdowns related to paging parts of the
    heap between physical and virtual memory; depending on memory access
    patterns, that could happen frequently and be very costly.

    On the other hand, some versions of the Java VM have problems with large
    amounts of heap memory. One would hope that such problems have been
    resolved by now, but you could anyway try reducing the heap memory to about
    1.5G. That's close to the maximum that you can safely use with some VM
    implementations.

    I certainly agree that puppetdb has a serious problem if it requires such
    large amounts of memory to handle resource collections with a few thousand
    elements. If roughly 3000 (at ~30 resources * ~100 nodes) exported
    resources require in excess of 3GB then that's in the vicinity of 1MB per
    resource -- absurd!

    On the other hand, that's *so* absurd that I suspect something else is
    going on here. It smacks of a combinatoric problem, either in your
    manifests or in the puppetdb implementation. There aren't enough data
    points to be sure, but your run times could be scaling with the square of
    the number of nodes. Is there any chance that the number of exported
    resources is scaling the same way? That would explain both time and memory
    consumption.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/25e0ER5_C4gJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Daniel at Jan 24, 2013 at 11:25 am
    Seems that chaining exported resources might not be too efficient and
    produces lots of data that could be the reason for puppetdb crashing.

    The culprits being these two lines in two manifest files:

    ./nsca/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
    ./nrpe/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>

    replacing them with unchained:

    File <<| tag == $get_tag |>>
    Nagios_host <<| tag == $get_tag |>>

    causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10
    mins, which is acceptable.

    This is tracked under bug #18804

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/S-md_OBZEF8J.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jcbollinger at Jan 24, 2013 at 2:41 pm

    On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:
    Seems that chaining exported resources might not be too efficient and
    produces lots of data that could be the reason for puppetdb crashing.

    The culprits being these two lines in two manifest files:

    ./nsca/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
    ./nrpe/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
    And there may be your combinatorial problem. Supposing that the tags are
    not specific to the exporting node, the numbers of Files and Nagios_hosts
    increase linearly with the number of nodes, but the number of relationships
    among them increases with the square of the number of nodes.

    replacing them with unchained:

    File <<| tag == $get_tag |>>
    Nagios_host <<| tag == $get_tag |>>

    causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10
    mins, which is acceptable.
    Do you in fact need the order of application constraints that the chaining
    declared? All of them?

    This is an area that seems to give people problems. On one hand, Puppet
    provides a couple of mechanisms that allow people to compactly declare
    large numbers of ordering relationships. On the other hand, people
    sometimes lose sight of the distinction between functional dependency and
    application-order dependency. Either one of these can cause trouble, as
    unneeded declarations of any kind slow Puppet and make it consume more
    resources. They also make you manifests more susceptible to dependency
    cycles. The combination is worse, however, because the number of unneeded
    relationships produced can be multiplicative.

    Depending on what relationships you actually need, you have various options:

    1) If it doesn't actually matter whether Puppet syncs given Files before or
    after it syncs any particular Nagios_host, then you never needed any
    relationships at all, and simply removing the chaining as you did is the
    best solution.

    2) If specific Files must be synced before specific Nagios_hosts, then you
    should express those relationships and not any others. In particular, if
    you only need relationships among Files and Nagios_hosts exported by the
    same node, then you should declare the relationships on the exporting side
    instead of on the collecting side. (And if they always come in such pairs
    then perhaps you should wrap then in a defined type and export that
    instead).

    3) If you really need something along the lines that the chaining expressed
    -- all Files with a given tag synced before all Nagios_hosts bearing that
    tag -- then you can break the combinatorial problem by introducing an
    artificial barrier resource for each tag:

    define mymodule::barrier() {
    # no content
    }

    mymodule::barrier { $get_tag: }

    File<<| tag == $get_tag |>> -> Mymodule::Barrier[ $get_tag ] -> Nagios_host
    <<| tag == $get_tag |>>

    That gives you (number(Files) + number(Nagios_host)) relationships for each
    tag instead of (number(Files) * number(Nagios_host)) relationships.

    Of course, you can use any single resource as a barrier; it doesn't need to
    be of a defined type. In some cases there might be a real resource that
    naturally fills the role. If you have Puppet's stdlib module then the
    Anchor resource would be a good fit; it is intended for a similar purpose
    anyway.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/2_AvVt-Qm4oJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ken Barber at Jan 24, 2013 at 3:56 pm
    I think John is on to something here, Daniel - I haven't seen your
    full content yet, but are you creating a file for each nagios_host, so
    that you can use that as the 'target'? Thus creating a single file for
    each nagios_host entry?

    If this is the case, the John is spot-on ... and you're creating more
    relationships than necessary. You probably want one file to one
    nagios_* resource, and thus one 'edge' for each, not this many-to-many
    case you seem to be creating at collection time.

    Of course I'm just guessing, not having seen your content.
    On Thu, Jan 24, 2013 at 2:41 PM, jcbollinger wrote:
    On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:

    Seems that chaining exported resources might not be too efficient and
    produces lots of data that could be the reason for puppetdb crashing.

    The culprits being these two lines in two manifest files:

    ./nsca/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag ==
    $get_tag |>>
    ./nrpe/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag ==
    $get_tag |>>


    And there may be your combinatorial problem. Supposing that the tags are
    not specific to the exporting node, the numbers of Files and Nagios_hosts
    increase linearly with the number of nodes, but the number of relationships
    among them increases with the square of the number of nodes.
    replacing them with unchained:

    File <<| tag == $get_tag |>>
    Nagios_host <<| tag == $get_tag |>>

    causes it to run even with 1GB for puppetdb (still a 16GB vm) in under 10
    mins, which is acceptable.

    Do you in fact need the order of application constraints that the chaining
    declared? All of them?

    This is an area that seems to give people problems. On one hand, Puppet
    provides a couple of mechanisms that allow people to compactly declare large
    numbers of ordering relationships. On the other hand, people sometimes lose
    sight of the distinction between functional dependency and application-order
    dependency. Either one of these can cause trouble, as unneeded declarations
    of any kind slow Puppet and make it consume more resources. They also make
    you manifests more susceptible to dependency cycles. The combination is
    worse, however, because the number of unneeded relationships produced can be
    multiplicative.

    Depending on what relationships you actually need, you have various options:

    1) If it doesn't actually matter whether Puppet syncs given Files before or
    after it syncs any particular Nagios_host, then you never needed any
    relationships at all, and simply removing the chaining as you did is the
    best solution.

    2) If specific Files must be synced before specific Nagios_hosts, then you
    should express those relationships and not any others. In particular, if
    you only need relationships among Files and Nagios_hosts exported by the
    same node, then you should declare the relationships on the exporting side
    instead of on the collecting side. (And if they always come in such pairs
    then perhaps you should wrap then in a defined type and export that
    instead).

    3) If you really need something along the lines that the chaining expressed
    -- all Files with a given tag synced before all Nagios_hosts bearing that
    tag -- then you can break the combinatorial problem by introducing an
    artificial barrier resource for each tag:

    define mymodule::barrier() {
    # no content
    }

    mymodule::barrier { $get_tag: }

    File<<| tag == $get_tag |>> -> Mymodule::Barrier[ $get_tag ] -> Nagios_host
    <<| tag == $get_tag |>>

    That gives you (number(Files) + number(Nagios_host)) relationships for each
    tag instead of (number(Files) * number(Nagios_host)) relationships.

    Of course, you can use any single resource as a barrier; it doesn't need to
    be of a defined type. In some cases there might be a real resource that
    naturally fills the role. If you have Puppet's stdlib module then the
    Anchor resource would be a good fit; it is intended for a similar purpose
    anyway.


    John


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/2_AvVt-Qm4oJ.

    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Daniel at Jan 25, 2013 at 12:14 pm

    On Thursday, January 24, 2013 2:41:53 PM UTC, jcbollinger wrote:
    On Thursday, January 24, 2013 5:24:58 AM UTC-6, Daniel wrote:

    Seems that chaining exported resources might not be too efficient and
    produces lots of data that could be the reason for puppetdb crashing.

    The culprits being these two lines in two manifest files:

    ./nsca/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
    ./nrpe/server.pp: #File <<| tag == $get_tag |>> -> Nagios_host <<| tag == $get_tag |>>
    And there may be your combinatorial problem. Supposing that the tags are
    not specific to the exporting node, the numbers of Files and Nagios_hosts
    increase linearly with the number of nodes, but the number of relationships
    among them increases with the square of the number of nodes.
    Do you in fact need the order of application constraints that the chaining
    declared? All of them?
    I don't need all of them. The reason I added it was to fix problems with
    files not being created when puppet tried to realize services, etc., thus
    producing an incorrect nagios setup. It worked in my small environment and
    only gave me a pause after days of trying to figure out what the hell is
    going on in the larger one.

    Looks like I didn't understand implications of this configuration, but to
    be honest I haven't found much in terms of documentation on chaining
    exported resources, potential implications and problems. Now I know, so
    "I've learnt something today" :).

    One suggestion - I would have a "WFT?" moment a lot sooner if --summarize
    displayed a number of relationships generated by my crazy config. I don't
    know if it's possible/easy to implement, but may be worth having a look at.

    Depending on what relationships you actually need, you have various options:
    1) If it doesn't actually matter whether Puppet syncs given Files before
    or after it syncs any particular Nagios_host, then you never needed any
    relationships at all, and simply removing the chaining as you did is the
    best solution.
    As stated above, I've added it after failures, so I do need some sort of
    relationships.


    2) If specific Files must be synced before specific Nagios_hosts, then you
    should express those relationships and not any others. In particular, if
    you only need relationships among Files and Nagios_hosts exported by the
    same node, then you should declare the relationships on the exporting side
    instead of on the collecting side. (And if they always come in such pairs
    then perhaps you should wrap then in a defined type and export that
    instead).
    That's the most likely approach I'm going to implement in the "next
    iteration" of this module.


    3) If you really need something along the lines that the chaining expressed
    -- all Files with a given tag synced before all Nagios_hosts bearing that
    tag -- then you can break the combinatorial problem by introducing an
    artificial barrier resource for each tag:
    I most likely don't need this, but will have it in mind for future
    experiments with exported resources.

    Thanks for your comprehensive answer.


    Regards,
    Daniel

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedJan 21, '13 at 4:13p
activeJan 25, '13 at 12:14p
posts16
users7
websitepuppetlabs.com

People

Translate

site design / logo © 2021 Grokbase