[Puppet Users] hiera command line performance

----- Original Message -----
From: "Paul Colby" <pco...@gmail.com <javascript:>>
To: puppet...@googlegroups.com <javascript:>
Sent: Monday, August 13, 2012 6:22:10 PM
Subject: [Puppet Users] hiera command line performance

Hi everyone,


The hiera command line program (version 0.3.0) is currently taking
around 2 seconds per invocation. This seems quite slow to me, but is
probably not an issue for most users.

We have a number of legacy machines that for one reason or another
have not been puppetised yet, and probably won't be anytime soon, so
I have a somewhat clever script that, for each of these hosts, uses
the hiera command like utility to export a few puppet-templated
documents just as puppet+hiera does for our puppeted server. This
works very nicely, but involves ~250 hiera lookups per host, which
at 2 seconds per lookup, is taking more than 8 minutes per host (and
using near 100% of a CPU core the entire time). As we have ~40 such
hosts, that's over 5 hours with no concurrency (we do use some
concurrency, but since each invocation uses near 100% of a CPU core
each, there's no point running more simultaneous hiera processes
than the number of CPUs, and even then, we need some reserved for
other services).


So, is there some way I can make this export run faster? The bulk of
the time seems to be spent in initialising the hiera command line
program, not the actual data lookup (for example, if I fail to
provide the necessary identity / scope files, it still takes 2
seconds to get around to returning an error, but if I fail to
provide a config file, it returns instantaneously with error). So,
I'd either like to make the program load / initialise much faster,
or be able to query multiple values for a single hiera command line
invocation.


Also, is there a more recent version of hiera that would be faster?
(I'm a little unclear about hiera versioning - we're running 0.3.0,
but I don't know if that's recent or old).


Any suggestions / tips would be greatly appreciated :)


Thanks.


Paul.

--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/YTZZ7FOshJkJ .
To post to this group, send email to puppet...@googlegroups.com<javascript:>.
To unsubscribe from this group, send email to
puppet-users...@googlegroups.com <javascript:>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wednesday, August 15, 2012 2:45:06 AM UTC+10, R.I. Pienaar wrote:

hello,
2 seconds seem quite slow though i guess it depends on the size of your
yamls and so forth, sounds like yours are big and complex. I'd hope the
bulk of the time is spent on starting/stopping ruby each time.

A better option might be to consider writing your script in ruby rather
than bash so you can then do your lookup that way reusing the single
hiera instance and avoid the ruby start/stop cost

So a simple test on my setup doing:

for i in {1..100}
do
hiera syslocation ::location=foo
done

this takes 30 seconds, ages. Here is the same in Ruby reusing the
class and using a single interpreter:

h = Hiera.new(:config => "/etc/hiera.yaml")

Benchmark.measure do
100.times do
puts h.lookup("syslocation", "", {"::location" => "foo"})
end
end

this takes 0.1 second for 100 lookups. If you know a bit of ruby this
would be a good approach for you even if you just write your script to
configure a single machine in ruby and run it 40 times only