FAQ
Hi everyone,

The hiera command line program (version 0.3.0) is currently taking around 2
seconds per invocation. This seems quite slow to me, but is probably not
an issue for most users.

We have a number of legacy machines that for one reason or another have not
been puppetised yet, and probably won't be anytime soon, so I have a
somewhat clever script that, for each of these hosts, uses the hiera command
like utility to export a few puppet-templated documents just as
puppet+hiera does for our puppeted server. This works very nicely, but
involves ~250 hiera lookups per host, which at 2 seconds per lookup, is
taking more than 8 minutes per host (and using near 100% of a CPU core the
entire time). As we have ~40 such hosts, that's over 5 hours with no
concurrency (we do use some concurrency, but since each invocation uses
near 100% of a CPU core each, there's no point running more simultaneous
hiera processes than the number of CPUs, and even then, we need some
reserved for other services).

So, is there some way I can make this export run faster? The bulk of the
time seems to be spent in initialising the hiera command line program, not
the actual data lookup (for example, if I fail to provide the necessary
identity / scope files, it still takes 2 seconds to get around to returning
an error, but if I fail to provide a config file, it returns
instantaneously with error). So, I'd either like to make the program load
/ initialise much faster, or be able to query multiple values for a single
hiera command line invocation.

Also, is there a more recent version of hiera that would be faster? (I'm a
little unclear about hiera versioning - we're running 0.3.0, but I don't
know if that's recent or old).

Any suggestions / tips would be greatly appreciated :)

Thanks.

Paul.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • R.I.Pienaar at Aug 14, 2012 at 4:45 pm
    hello,
    ----- Original Message -----
    From: "Paul Colby" <pcolby@gmail.com>
    To: puppet-users@googlegroups.com
    Sent: Monday, August 13, 2012 6:22:10 PM
    Subject: [Puppet Users] hiera command line performance

    Hi everyone,


    The hiera command line program (version 0.3.0) is currently taking
    around 2 seconds per invocation. This seems quite slow to me, but is
    probably not an issue for most users.
    2 seconds seem quite slow though i guess it depends on the size of your
    yamls and so forth, sounds like yours are big and complex. I'd hope the
    bulk of the time is spent on starting/stopping ruby each time.

    A better option might be to consider writing your script in ruby rather
    than bash so you can then do your lookup that way reusing the single
    hiera instance and avoid the ruby start/stop cost

    So a simple test on my setup doing:

    for i in {1..100}
    do
    hiera syslocation ::location=foo
    done

    this takes 30 seconds, ages. Here is the same in Ruby reusing the
    class and using a single interpreter:

    h = Hiera.new(:config => "/etc/hiera.yaml")

    Benchmark.measure do
    100.times do
    puts h.lookup("syslocation", "", {"::location" => "foo"})
    end
    end

    this takes 0.1 second for 100 lookups. If you know a bit of ruby this
    would be a good approach for you even if you just write your script to
    configure a single machine in ruby and run it 40 times only

    We have a number of legacy machines that for one reason or another
    have not been puppetised yet, and probably won't be anytime soon, so
    I have a somewhat clever script that, for each of these hosts, uses
    the hiera command like utility to export a few puppet-templated
    documents just as puppet+hiera does for our puppeted server. This
    works very nicely, but involves ~250 hiera lookups per host, which
    at 2 seconds per lookup, is taking more than 8 minutes per host (and
    using near 100% of a CPU core the entire time). As we have ~40 such
    hosts, that's over 5 hours with no concurrency (we do use some
    concurrency, but since each invocation uses near 100% of a CPU core
    each, there's no point running more simultaneous hiera processes
    than the number of CPUs, and even then, we need some reserved for
    other services).


    So, is there some way I can make this export run faster? The bulk of
    the time seems to be spent in initialising the hiera command line
    program, not the actual data lookup (for example, if I fail to
    provide the necessary identity / scope files, it still takes 2
    seconds to get around to returning an error, but if I fail to
    provide a config file, it returns instantaneously with error). So,
    I'd either like to make the program load / initialise much faster,
    or be able to query multiple values for a single hiera command line
    invocation.


    Also, is there a more recent version of hiera that would be faster?
    (I'm a little unclear about hiera versioning - we're running 0.3.0,
    but I don't know if that's recent or old).


    Any suggestions / tips would be greatly appreciated :)


    Thanks.


    Paul.

    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Paul Colby at Aug 15, 2012 at 1:03 am
    Thanks R.I. Pienaar, that's exactly the kind of suggestion I was hoping
    for! :)

    I haven't written any Ruby for ~9 years, but as I regularly develop in
    several other languages (primarily C++), I'll enjoy getting my hands dirty
    with Ruby again.

    And thanks for including a basic single-interpreter example - that will
    help me get going a lot quicker!

    Thanks,

    Paul.
    On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:

    hello,
    ----- Original Message -----
    From: "Paul Colby" <pco...@gmail.com <javascript:>>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Monday, August 13, 2012 6:22:10 PM
    Subject: [Puppet Users] hiera command line performance

    Hi everyone,


    The hiera command line program (version 0.3.0) is currently taking
    around 2 seconds per invocation. This seems quite slow to me, but is
    probably not an issue for most users.
    2 seconds seem quite slow though i guess it depends on the size of your
    yamls and so forth, sounds like yours are big and complex. I'd hope the
    bulk of the time is spent on starting/stopping ruby each time.

    A better option might be to consider writing your script in ruby rather
    than bash so you can then do your lookup that way reusing the single
    hiera instance and avoid the ruby start/stop cost

    So a simple test on my setup doing:

    for i in {1..100}
    do
    hiera syslocation ::location=foo
    done

    this takes 30 seconds, ages. Here is the same in Ruby reusing the
    class and using a single interpreter:

    h = Hiera.new(:config => "/etc/hiera.yaml")

    Benchmark.measure do
    100.times do
    puts h.lookup("syslocation", "", {"::location" => "foo"})
    end
    end

    this takes 0.1 second for 100 lookups. If you know a bit of ruby this
    would be a good approach for you even if you just write your script to
    configure a single machine in ruby and run it 40 times only

    We have a number of legacy machines that for one reason or another
    have not been puppetised yet, and probably won't be anytime soon, so
    I have a somewhat clever script that, for each of these hosts, uses
    the hiera command like utility to export a few puppet-templated
    documents just as puppet+hiera does for our puppeted server. This
    works very nicely, but involves ~250 hiera lookups per host, which
    at 2 seconds per lookup, is taking more than 8 minutes per host (and
    using near 100% of a CPU core the entire time). As we have ~40 such
    hosts, that's over 5 hours with no concurrency (we do use some
    concurrency, but since each invocation uses near 100% of a CPU core
    each, there's no point running more simultaneous hiera processes
    than the number of CPUs, and even then, we need some reserved for
    other services).


    So, is there some way I can make this export run faster? The bulk of
    the time seems to be spent in initialising the hiera command line
    program, not the actual data lookup (for example, if I fail to
    provide the necessary identity / scope files, it still takes 2
    seconds to get around to returning an error, but if I fail to
    provide a config file, it returns instantaneously with error). So,
    I'd either like to make the program load / initialise much faster,
    or be able to query multiple values for a single hiera command line
    invocation.


    Also, is there a more recent version of hiera that would be faster?
    (I'm a little unclear about hiera versioning - we're running 0.3.0,
    but I don't know if that's recent or old).


    Any suggestions / tips would be greatly appreciated :)


    Thanks.


    Paul.

    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
    To post to this group, send email to puppet...@googlegroups.com<javascript:>.
    To unsubscribe from this group, send email to
    puppet-users...@googlegroups.com <javascript:>.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/WQV-NVyOHXgJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Paul Colby at Aug 24, 2012 at 10:03 am
    Thanks again R.I. Pienaar.

    I just got time to get back to this, I've implemented a basic ruby script
    to export the hiera data as you've suggested - works extremely well!! :)

    Interestingly, the performance issue seems to be with puppet specifically,
    not hiera.

    Here's a slightly cut-down version of my export script:

    #!/usr/bin/ruby

    require 'rubygems'
    require 'hiera'
    #require 'puppet'

    # Load the "facts" for the host to export settings for.
    if ARGV.length != 1
    puts "Usage: #{File.basename($PROGRAM_NAME)} /path/to/host-facts.yaml"
    exit
    end
    scope = YAML.load_file(ARGV.first)
    scope['calling_module'] = 'settings'

    # Fetch all settings for this host from hiera.
    hiera = Hiera.new(:config => '/etc/puppet/hiera.yaml')
    settings = hiera.lookup('settings', '', scope, '', :array)
    settings.each do |setting|
    puts "#{setting}=#{hiera.lookup(setting, '', scope)}"
    end

    Note that the "require 'puppet'" line is commented out... code examples on
    the net use that statement, but the scripts runs fine both with and without
    it.

    The interesting thing is, without that require, the script is nice and
    quick, but with it, it is 2.5 times slower.

    Example times:

    With "require puppet"
    real 0m2.448s
    user 0m2.123s
    sys 0m0.323s

    Without "require puppet"
    real 0m0.917s
    user 0m0.808s
    sys 0m0.098s

    Anyway, I'm very happy with the result! The inner "*.each" call at the end
    means I'm calling this script ~40 times, instead of the (slower) hiera
    command line program ~10,000 times. Much faster, and lower CPU usage too :)

    Thanks again.

    Paul.
    On Wednesday, August 15, 2012 10:55:15 AM UTC+10, Paul Colby wrote:

    Thanks R.I. Pienaar, that's exactly the kind of suggestion I was hoping
    for! :)

    I haven't written any Ruby for ~9 years, but as I regularly develop in
    several other languages (primarily C++), I'll enjoy getting my hands dirty
    with Ruby again.

    And thanks for including a basic single-interpreter example - that will
    help me get going a lot quicker!

    Thanks,

    Paul.
    On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:

    hello,
    ----- Original Message -----
    From: "Paul Colby" <pco...@gmail.com>
    To: puppet...@googlegroups.com
    Sent: Monday, August 13, 2012 6:22:10 PM
    Subject: [Puppet Users] hiera command line performance

    Hi everyone,


    The hiera command line program (version 0.3.0) is currently taking
    around 2 seconds per invocation. This seems quite slow to me, but is
    probably not an issue for most users.
    2 seconds seem quite slow though i guess it depends on the size of your
    yamls and so forth, sounds like yours are big and complex. I'd hope the
    bulk of the time is spent on starting/stopping ruby each time.

    A better option might be to consider writing your script in ruby rather
    than bash so you can then do your lookup that way reusing the single
    hiera instance and avoid the ruby start/stop cost

    So a simple test on my setup doing:

    for i in {1..100}
    do
    hiera syslocation ::location=foo
    done

    this takes 30 seconds, ages. Here is the same in Ruby reusing the
    class and using a single interpreter:

    h = Hiera.new(:config => "/etc/hiera.yaml")

    Benchmark.measure do
    100.times do
    puts h.lookup("syslocation", "", {"::location" => "foo"})
    end
    end

    this takes 0.1 second for 100 lookups. If you know a bit of ruby this
    would be a good approach for you even if you just write your script to
    configure a single machine in ruby and run it 40 times only

    We have a number of legacy machines that for one reason or another
    have not been puppetised yet, and probably won't be anytime soon, so
    I have a somewhat clever script that, for each of these hosts, uses
    the hiera command like utility to export a few puppet-templated
    documents just as puppet+hiera does for our puppeted server. This
    works very nicely, but involves ~250 hiera lookups per host, which
    at 2 seconds per lookup, is taking more than 8 minutes per host (and
    using near 100% of a CPU core the entire time). As we have ~40 such
    hosts, that's over 5 hours with no concurrency (we do use some
    concurrency, but since each invocation uses near 100% of a CPU core
    each, there's no point running more simultaneous hiera processes
    than the number of CPUs, and even then, we need some reserved for
    other services).


    So, is there some way I can make this export run faster? The bulk of
    the time seems to be spent in initialising the hiera command line
    program, not the actual data lookup (for example, if I fail to
    provide the necessary identity / scope files, it still takes 2
    seconds to get around to returning an error, but if I fail to
    provide a config file, it returns instantaneously with error). So,
    I'd either like to make the program load / initialise much faster,
    or be able to query multiple values for a single hiera command line
    invocation.


    Also, is there a more recent version of hiera that would be faster?
    (I'm a little unclear about hiera versioning - we're running 0.3.0,
    but I don't know if that's recent or old).


    Any suggestions / tips would be greatly appreciated :)


    Thanks.


    Paul.

    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
    To post to this group, send email to puppet...@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users...@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zS2noBFW0ZoJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • R.I.Pienaar at Aug 24, 2012 at 10:31 am

    ----- Original Message -----
    From: "Paul Colby" <pcolby@gmail.com>
    To: puppet-users@googlegroups.com
    Sent: Friday, August 24, 2012 11:03:51 AM
    Subject: Re: [Puppet Users] hiera command line performance

    Thanks again R.I. Pienaar.
    awesome, glad you could sort it out :)

    pretty interesting that just requiring puppet slows it down that much

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedAug 14, '12 at 1:22a
activeAug 24, '12 at 10:31a
posts5
users2
websitepuppetlabs.com

2 users in discussion

Paul Colby: 3 posts R.I.Pienaar: 2 posts

People

Translate

site design / logo © 2022 Grokbase