FAQ
are we having problem with defines in puppet 3.0.1/3.0.2?
the more and more we use defines in our repo, the more lagish puppetmaster
becomes.

today i've made some synthetic checks with our lb setup, which consists of
121 defines, each of them wraps around 4 another defines (nagios, haproxy,
stunnel configs).
results are catastrophic.

even more, we started to use ruby 1.9.3, as 1.8.7 is becoming obsolete and
it's adding a more lag.

i've reduced define numbers and observed that compile time reduces
significantly each time.
additionally i've inlined code from each define that we source from our lb
define, and it also speeds up compile a lot.

i've compiled with:
time /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/bin/puppet master
--compile lb2
and
/usr/bin/ruby /usr/bin/puppet master --compile lb2

actual times are here:
https://github.com/asquelt/puppet-defines-wtf/blob/master/BENCHMARK
manifest is here:
https://github.com/asquelt/puppet-defines-wtf/blob/master/manifests/projects/dc1-lb.pp
catalogs are also in repo: https://github.com/asquelt/puppet-defines-wtf

you can see that i've decorated everything with double colon to make sure
this is not related to #17335

is it just me?
do you have any workarounds?
should i file a bug?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/yJhfP5lqsqYJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Jcbollinger at Jan 25, 2013 at 3:07 pm

    On Thursday, January 24, 2013 1:29:32 PM UTC-6, asq wrote:
    are we having problem with defines in puppet 3.0.1/3.0.2?
    the more and more we use defines in our repo, the more lagish puppetmaster
    becomes.

    today i've made some synthetic checks with our lb setup, which consists of
    121 defines, each of them wraps around 4 another defines (nagios, haproxy,
    stunnel configs).
    results are catastrophic.

    even more, we started to use ruby 1.9.3, as 1.8.7 is becoming obsolete and
    it's adding a more lag.

    i've reduced define numbers and observed that compile time reduces
    significantly each time.
    additionally i've inlined code from each define that we source from our lb
    define, and it also speeds up compile a lot.

    i've compiled with:
    time /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/bin/puppet master
    --compile lb2
    and
    /usr/bin/ruby /usr/bin/puppet master --compile lb2

    actual times are here:
    https://github.com/asquelt/puppet-defines-wtf/blob/master/BENCHMARK
    manifest is here:
    https://github.com/asquelt/puppet-defines-wtf/blob/master/manifests/projects/dc1-lb.pp
    catalogs are also in repo: https://github.com/asquelt/puppet-defines-wtf

    you can see that i've decorated everything with double colon to make sure
    this is not related to #17335

    is it just me?
    do you have any workarounds?
    should i file a bug?

    To be clear, it looks like you are talking about counts of defined type *
    instances*, not counts of distinct defined types. Is that right?

    I am inclined to suspect that the problem is not defined types in general,
    but rather the specific defined types you are using. Indeed, your data
    seem to support that inasmuch as inlining definitions doesn't cut your
    compilation times very much.

    I can think of several things might cause long compilation times:

    - evaluating many templates, especially complex ones
    - making many hiera lookups if your hierarchy definition uses
    interpolated non-global variables
    - repetition of anything requiring network access, such as syncing files
    with a version-control repository, accessing files on a remote file system,
    or performing name resolutions.
    - calling generate() a lot

    That's not an exhaustive list.

    Anyway, I think filing a bug now would be premature. It may be that there
    is a bona fide performance problem, but if so then I don't think it has
    been pinpointed yet.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Asq at Jan 25, 2013 at 3:29 pm
    W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger
    napisał:

    To be clear, it looks like you are talking about counts of defined type *
    instances*, not counts of distinct defined types. Is that right?
    yes. it's test::rule() and test::rule_inline() in linked manifest.

    I am inclined to suspect that the problem is not defined types in general,
    but rather the specific defined types you are using. Indeed, your data
    seem to support that inasmuch as inlining definitions doesn't cut your
    compilation times very much.
    we're talking about 5m33s and 4m13s difference in compile. this *is* a lot.

    I can think of several things might cause long compilation times:

    - evaluating many templates, especially complex ones
    - making many hiera lookups if your hierarchy definition uses
    interpolated non-global variables
    - repetition of anything requiring network access, such as syncing
    files with a version-control repository, accessing files on a remote file
    system, or performing name resolutions.
    - calling generate() a lot

    That's not an exhaustive list.
    please check define rule_inline() in linked pp file - it's nothing fancy.

    - 3 static files
    - 1 erb (35 lines with 5 variables substitution and 1 if/else clause)
    - 1 hiera call
    - 1 exported resource
    - 4 calls to custom type (firewall)

    no generates or other puppetmaster functions, no network stuff, no
    dns-related stuff or anything i haven't already hashed-out to test if it
    will affect compile times significantly.

    Anyway, I think filing a bug now would be premature. It may be that there
    is a bona fide performance problem, but if so then I don't think it has
    been pinpointed yet.
    i filed a bug #18880 anyway, as i sense it might be some design error in
    puppet - i can't really see why number of define() calls should relate to
    compile times. as they are not "real" resources, there should be no reason
    for them to be included in catalog, which also might relate to apply times.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Asq at Jan 25, 2013 at 4:15 pm
    W dniu piątek, 25 stycznia 2013 16:29:00 UTC+1 użytkownik asq napisał:
    W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger
    napisał:
    please check define rule_inline() in linked pp file - it's nothing fancy.
    - 3 static files
    - 1 erb (35 lines with 5 variables substitution and 1 if/else clause)
    - 1 hiera call
    - 1 exported resource
    - 4 calls to custom type (firewall)
    disabling hiera i shaved only 11 seconds on ruby rvm (6 seconds on stock
    ruby).
    disabling erb template made no measurable difference.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jcbollinger at Jan 28, 2013 at 3:49 pm

    On Friday, January 25, 2013 9:29:00 AM UTC-6, asq wrote:
    W dniu piątek, 25 stycznia 2013 16:07:43 UTC+1 użytkownik jcbollinger
    napisał:

    To be clear, it looks like you are talking about counts of defined type *
    instances*, not counts of distinct defined types. Is that right?
    yes. it's test::rule() and test::rule_inline() in linked manifest.

    I am inclined to suspect that the problem is not defined types in
    general, but rather the specific defined types you are using. Indeed, your
    data seem to support that inasmuch as inlining definitions doesn't cut your
    compilation times very much.
    we're talking about 5m33s and 4m13s difference in compile. this *is* a lot.

    The version using more defined-type instances has more total resources.
    The time per resource is very nearly the same. Your compile times
    certainly will scale with the total number of resources declared.

    Do not confuse defined types with macros. They are bona fide resource
    types, functioning like native types in most ways.


    I can think of several things might cause long compilation times:

    - evaluating many templates, especially complex ones
    - making many hiera lookups if your hierarchy definition uses
    interpolated non-global variables
    - repetition of anything requiring network access, such as syncing
    files with a version-control repository, accessing files on a remote file
    system, or performing name resolutions.
    - calling generate() a lot

    That's not an exhaustive list.
    please check define rule_inline() in linked pp file - it's nothing fancy.

    Fanciness is not the question. Even very simple manifests can make
    declarations that are costly to compile.


    - 3 static files
    - 1 erb (35 lines with 5 variables substitution and 1 if/else clause)

    The erb evaluation is probably more costly than everything else about that
    particular File declaration, combined. Especially so in that it requires
    (in that particular case) up to four separate probes of the file system
    just to find the template file.


    - 1 hiera call

    I believe hiera caches data to the extent that it can do, but depending on
    how your hierarchy is defined and how your data are arranged, it could
    still be that each one of those hiera() calls involves filesystem probes
    and possibly reading and parsing YAML. That's not particularly cheap even
    if your data are simple.


    - 1 exported resource

    I must have overlooked that, but it's not especially cheap, either,
    inasmuch as it involves connecting to and querying a DB. It's practically
    expensive if the underlying DB is not local, as that then would involve
    network activity.


    - 4 calls to custom type (firewall)

    Which are probably cheap(ish), but that's not certain.


    no generates or other puppetmaster functions, no network stuff, no
    dns-related stuff or anything i haven't already hashed-out to test if it
    will affect compile times significantly.

    Anyway, I think filing a bug now would be premature. It may be that there
    is a bona fide performance problem, but if so then I don't think it has
    been pinpointed yet.
    i filed a bug #18880 anyway, as i sense it might be some design error in
    puppet - i can't really see why number of define() calls should relate to
    compile times. as they are not "real" resources, there should be no reason
    for them to be included in catalog, which also might relate to apply times.
    Definition instances certainly are real resources in most ways that
    matter. In any event, your compilation times even for the "inline" version
    of your manifests seem unreasonably high, which is why I suggested that you
    look at other possible performance drags.

    In addition to the manifest-specific issues I discussed above, you should
    also look at how your master is provisioned and loaded. Puppet is a bit of
    a memory hog, so if the available physical RAM is low then Puppet may
    perform a lot of expensive swaps between physical and virtual RAM. If the
    master is sharing hardware with other services (even other masters in, for
    example, a passenger setup), then contention for the disk subsystem could
    slow everything, especially if you have SATA (or worse) disks instead of
    SCSI. There are many other such possibilities.

    On the other hand, I don't mean to suggest that declaration of defined-type
    instances is free or cheap. As I said, defined types are not macros, but
    rather bona fide resource types. Declaring a defined-type instance
    involves more work than simply making all the declarations that appear in
    the definition body.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Asq at Jan 28, 2013 at 4:04 pm
    W dniu poniedziałek, 28 stycznia 2013 16:49:38 UTC+1 użytkownik jcbollinger
    napisał:
    - 1 erb (35 lines with 5 variables substitution and 1 if/else clause)

    The erb evaluation is probably more costly than everything else about that
    particular File declaration, combined. Especially so in that it requires
    (in that particular case) up to four separate probes of the file system
    just to find the template file.
    i replaced this erb with string "foo" and it showed sub-second differences,
    not really measurable for 2000 resources.

    - 1 hiera call

    I believe hiera caches data to the extent that it can do, but depending on
    how your hierarchy is defined and how your data are arranged, it could
    still be that each one of those hiera() calls involves filesystem probes
    and possibly reading and parsing YAML. That's not particularly cheap even
    if your data are simple.
    disabling hiera shaves about 10 seconds on 2000 resources. not a big deal
    either.

    - 1 exported resource

    I must have overlooked that, but it's not especially cheap, either,
    inasmuch as it involves connecting to and querying a DB. It's practically
    expensive if the underlying DB is not local, as that then would involve
    network activity.
    disabling it shaves another 30 seconds on 2000 resources. nothing
    spectacular, considering that only adding 200-300 resources will make this
    time as high again.

    - 4 calls to custom type (firewall)
    Which are probably cheap(ish), but that's not certain.
    yes, hashing them out doesn't help much.

    In addition to the manifest-specific issues I discussed above, you should
    also look at how your master is provisioned and loaded. Puppet is a bit of
    a memory hog, so if the available physical RAM is low then Puppet may
    perform a lot of expensive swaps between physical and virtual RAM. If the
    master is sharing hardware with other services (even other masters in, for
    example, a passenger setup), then contention for the disk subsystem could
    slow everything, especially if you have SATA (or worse) disks instead of
    SCSI. There are many other such possibilities.
    i've tried to bind those problems to i/o but putting manifests on ramdisk
    doesn't help at all. manifests are still compiling very slow.

    i know that those times are high. but anyway i can't find a single "bit"
    that makes it so. i think puppet is very sensitible to the way its used.
    when you look at ruby 1.8/1.9 comparisions for the same manifests, you can
    clearly see that something worrying is happening, as newer ruby should be
    more optimized, but instead it runs significantly slower.

    on production we use passenger with no of puppetmaster processes equal to
    number of threads (and queue of excessive catalog requests on haproxy on
    front of it).

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Asq at Jan 28, 2013 at 4:08 pm
    W dniu poniedziałek, 28 stycznia 2013 17:04:51 UTC+1 użytkownik asq napisał:

    on production we use passenger with no of puppetmaster processes equal to
    number of threads (and queue of excessive catalog requests on haproxy on
    front of it).
    i mean - number of cores (8).

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jcbollinger at Jan 29, 2013 at 3:54 pm
    The bottom line is that I don't have an explanation for why your
    compilations are so slow. Sorry. I don't think it's anything specific to
    defined types, except inasmuch as defined type instances do have overhead
    comparable to that of native type instances. Perhaps the guys over on
    puppet-dev would have a suggestion about how you can profile catalog
    compilation to get an idea of where all the time is spent.

    On Monday, January 28, 2013 10:04:51 AM UTC-6, asq wrote:

    i know that those times are high. but anyway i can't find a single "bit"
    that makes it so. i think puppet is very sensitible to the way its used.
    when you look at ruby 1.8/1.9 comparisions for the same manifests, you can
    clearly see that something worrying is happening, as newer ruby should be
    more optimized, but instead it runs significantly slower.
    Of course Puppet's compilation speed is sensitive to the contents of your
    manifests, and more so than just to their raw size. It may be that you can
    substantially improve your performance if you can identify the problem
    area(s).

    On the other hand, I don't think it's at all safe to assume that a
    higher-version Ruby (or anything else) should yield greater performance.
    Performance optimization is certainly one development objective, but it is
    often at odds with new feature implementation and even with bug fixes.
    Especially with between Ruby 1.8 and Ruby 1.9 there are some deep changes
    in Ruby's string handling and character I/O, and these (among other things)
    might easily exact a performance penalty from applications, such as Puppet,
    that do a lot of text manipulation.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Asq at Jan 29, 2013 at 6:08 pm
    W dniu wtorek, 29 stycznia 2013 16:54:46 UTC+1 użytkownik jcbollinger
    napisał:

    The bottom line is that I don't have an explanation for why your
    compilations are so slow. Sorry. I don't think it's anything specific to
    defined types, except inasmuch as defined type instances do have overhead
    comparable to that of native type instances. Perhaps the guys over on
    puppet-dev would have a suggestion about how you can profile catalog
    compilation to get an idea of where all the time is spent.
    it's surely related to puppetmaster being slowed down by ruby gems, as i
    mentioned here:
    https://groups.google.com/d/topic/puppet-users/jpx9fNcFuMk/discussion
    it might be not only cause, but giving a puppet dedicated rvm gemset (just
    semver for puppet, json for pdb and gpgme for hiera) gave us some
    significant boost.

    another suspicious thing is how catalog is travelling over network. i think
    it should be compressed (lots of inlined templates) but for some reason it
    isn't (there is b64_zlib_yaml type, but there's no zlib_pson).

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedJan 24, '13 at 7:29p
activeJan 29, '13 at 6:08p
posts9
users2
websitepuppetlabs.com

2 users in discussion

Asq: 6 posts Jcbollinger: 3 posts

People

Translate

site design / logo © 2022 Grokbase