[Puppet Users] Hiera Questions: An array of :datadir: ?

Hi Dan,

That's my own terrible modifications to Hiera to support multiple data
stores of the same type. Code is here, but I warn you, I hacked in the
quickest way to get the functionality I wanted. I only modify
hiera-json, would be simple to do any others though:

https://github.com/lukebigum/hiera
https://github.com/lukebigum/hiera-json

As for use cases, a more elegant description is in this feature request:
http://projects.puppetlabs.com/issues/13954
Yes, but precedence is taken just like Hiera's natural hierarchy, so
data stores at the start of the array have higher priority. In my case
for example, "hiera_data_store" overrides "satellite_system_groups".
That was the the original idea of the hack - I had data coming from
several places - some JSON written by hand, some generated by scripts
(the 'satellite_system_groups' store is a dump of our old Red Hat
Satellite System Groups and 'rebirth_data_store' is procedurally
generated). I could have modified the generating scripts to read any
existing JSON and write out a merged file, but I was rushing and I
didn't think of that at the time ;-)

It has managed to keep a nice logical separation of types of Hiera keys
though and you can put your data stores in different locations, like
having a 'private' dir for password hashes that's not under revision
control. This leads to giving our Development team a Git repo that's the
lowest priority data store and allow them to specify their own keys to
configure their Dev servers. Any Admin still has the ability to
overwrite any key in an data store above it, so it stops them doing
anything silly like changing passwords, LDAP servers, etc, as these keys
will already be set by us (Admins).

-Luke

Thanks for responding, Luke.

This looks like a useful expansion of the hiera back-end, but as I am still fairly new to Puppet (and Python and Ruby for that matter), I think I will wait for this to be formally accepted and incorporated before trying it myself.

One parting question: Is the :hierarchy: repeated in each mamber of the :datadir: ? Logically, it should repeat, but I'd rather ask and be certain.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

Hi Dan,

Last week I was going to reply with "Sure does, here's an example
debug..." and I released that my code didn't work they way I thought it
did, I had the inner and outer loops around the wrong way ;-)

So after much more butchering of RIP's code I can happily report that
the :hierarchy: is searched as the outer loop, the :datadir: Array as
the inner loop. This means that first the %{fqdn} files are searched in
each member of :datadir:, then if nothing is found it moves on to
'%{lmax_env}_role' and so on down the hierarchy.

A (working!) example:

[root@gs2puppet01 ~]# /opt/ruby-enterprise/bin/hiera -c
/etc/puppet/hiera.yaml -y
/var/lib/puppet/yaml/facts/test.example.com.yaml -a collective --debug

DEBUG: Mon May 21 07:59:39 +0000 2012: Hiera JSON backend starting
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking up collective in JSON backend
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source
test.example.com
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/test.example.com.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/test.example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/test.example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/test.example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/test.example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _role
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/hiera_data_store/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/_role.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/_role.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source _server
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/hiera_data_store/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/_server.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/_server.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source example.com
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/private/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/satellite_system_groups/example.com.json,
skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/private/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/example.com.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Looking for data source common
DEBUG: Mon May 21 07:59:39 +0000 2012: Backend datadir for json is an
Array, multiple data dirs to search
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/private/common.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/hiera_data_store/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Cannot find datafile
/etc/puppet/environments/production/rebirth_data_store/common.json, skipping
DEBUG: Mon May 21 07:59:39 +0000 2012: Found datafile
/etc/puppet/environments/production/satellite_system_groups/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/hiera_data_store/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Attempting to read JSON from
/etc/puppet/environments/production/satellite_system_groups/common.json
DEBUG: Mon May 21 07:59:39 +0000 2012: Found key collective in file
/etc/puppet/environments/production/satellite_system_groups/common.json,
appending to Array
DEBUG: Mon May 21 07:59:39 +0000 2012: Lookup found answer ''
[""]