[Puppet Users] Puppet 3.0 and Hiera

----- Original Message -----
From: "Pieter van de Bruggen" <[email protected]> <snip>
* How should we integrate hiera_array() and hiera_hash() ?
* How should we integrate hiera ’s “default” and “override”
parameters?
* How should we handle overlaps between data supplied by Hiera
and data supplied in a parameterized class include?

On Fri, May 4, 2012 at 5:05 AM, R.I.Pienaar wrote:
I think the plan was that there would be a priority order as below:

- someone wrote in a manifest: class{"foo": something => something}
- an ENC supplied the values for something on the class foo
- someone did "include foo" or class{"foo": } this would consult hiera
- if hiera does not have an answer it would default to "default"

This gives people with more complex needs than those matched by hiera
the ability to use ENCs and get exactly the data model they desire as
well as people who do not want any magical data lookup for some class
the ability to hard code values.

And at the same time it lets module authors provide out of the box
defaults that work without placing a load on module users where they
might have to read every class and set hiera values for every argument
before they can use a class if all they wanted was the default out of
the box behaviour.

Does this sound sane?

On Fri, May 4, 2012 at 12:05 PM, Ohad Levy wrote:
Would be possible to define which function is called in this case? by
default that should be heira, but in case someone wants something else, I
think we should allow that?

---
hiera_var:
user:
value: 'my_value'
*module/params.pp*
$hiera_var = hiera_hash('hiera_var')
$sorted_var = sort(keys($hiera_var))
*module/template/sometemplate.pp*
<% scope.lookupvar('module::params::sorted_var').each do |item| -%><%
scope.lookupvar('hadoop::params::hiera_var')[item].each do |key,value| -%>
<%= user %> <%= key %> <%= value %>
<% end -%><% end -%>
On Thursday, May 3, 2012 10:05:14 AM UTC-7, Pieter van de Bruggen wrote:

Greetings,

As many of you may be aware, Hiera will be tightly integrated into Puppet
in the upcoming release of Puppet 3.0. As a final sanity check of this
work, I’d like to open our current plan for integration up for feedback.
This is particularly for feedback from existing Hiera users, but I hope to
solicit good feedback from those not using Hiera as well.

The problem, as it currently exists, is that Puppet (core) has no good
first-class mechanism for separating configuration data from manifests.
Everything from $faked_namespaces__in__variable_names to specialty
“params” classes have been tried, with varying degrees of success.

Hiera came along as another solution to this problem, and provided a
useful abstraction for hierarchical data lookup, but life as a plugin meant
that certain integrations were still difficult. Our aim in Puppet 3.0 is to
tighten up these integrations (making Hiera more useful), provide
first-class data separation solution (for those not already using Hiera),
and to provide a simple and safe migration for those currently using Hiera.

Here’s what’s new:

- Hiera data keys can be namespaced
- (e.g. 'dns::nameserver': '8.8.8.8')
- Namespaced data will automatically populate class parameters
- (e.g. 'dns::nameserver' will be automatically looked up for the
$nameserver parameter when you include dns)

Here’s what’s changed:

- The hiera-puppet module will no longer be required
- It should, however, still continue to work
- The heira() function (from the hiera-puppet module) will be
superceded by the lookup() function (in Puppet core)
- hiera_include() will not be ported; include now properly accept
arrays, making this redundant

Here’s what were still wondering about:

- How should we integrate hiera_array() and hiera_hash()?
- How should we integrate hiera’s “default” and “override” parameters?
- How should we handle overlaps between data supplied by Hiera and
data supplied in a parameterized class include?

If you’re interested in test-driving the new functionality for yourself,
checkout out the master branch of the Puppet repository on Github<https://github.com/puppetlabs/puppet>,
install Hiera (with gem install hiera) and make the following
configuration changes:

- Set data_binding_terminus to hiera
- If you have a Hiera configuration file, set hiera_config to point to
it.
- If you don’t, create a file in ${confdir}/hiera.yaml with these
contents:

:backend:

- yaml

:yaml:

:datadir: $confdir/data

:hierarchy:

- %{certname}

- %{environment}

- global

This sets up a default hierarchy that looks for values in ${confdir}/data,
first in the ${certname}.yml file, then in the ${environment}.yml file,
then in global.yml.

That should be it! Please, let us know if you're having trouble getting
started, or if you have questions, concerns, thoughts, or opinions about
any of this.

Thanks!

----- Original Message -----
From: "Jeremy Carroll" <[email protected]>
To: [email protected]
Sent: Friday, May 4, 2012 5:19:37 PM
Subject: [Puppet Users] Re: Puppet 3.0 and Hiera

We use Hiera backed Puppet here at Klout. Mainly for the reasons of
separating configuration data from puppet modules. We have had great
success, and I welcome this change to puppet core. Reminds me of
some of Chef's "Deep Merge" functionality. Here are a few of my
thoughts.


* Name spacing variable names is a welcome improvement. It will
help with variable name collisions we have experienced (Ex:
module1::hosts, module2::hosts). We currently use the
%{calling_class}, %{calling_module} method of scoping hiera
variables.
* I currently like the flexibility of defining my own order for
merging of variables in hiera.yaml. But I would like Puppet to
formalize the merge order so everybody can use the same style
when creating modules for reuse. Also have some methods to
specify overrides instead of merging from top to bottom for
hashes / arrays. Helps for configurations for systems that don't
want to inherit everything. Chef has a lot of work for Deep
Merging ordering that I believe Hiera would benefit from.

hieradata


---

hiera_var:
user:
value: 'my_value'
module/params.pp

$hiera_var = hiera_hash('hiera_var')
$sorted_var = sort(keys($hiera_var))
module/template/sometemplate.pp


<% scope.lookupvar('module::params::sorted_var').each do |item| -%><%
scope.lookupvar('hadoop::params::hiera_var')[item].each do
<%= user %> <%= key %> <%= value %>
<% end -%><% end -%>
On Thursday, May 3, 2012 10:05:14 AM UTC-7, Pieter van de Bruggen
wrote:





Greetings,

As many of you may be aware, Hiera will be tightly integrated into
Puppet in the upcoming release of Puppet 3.0. As a final sanity
check of this work, I’d like to open our current plan for
integration up for feedback. This is particularly for feedback from
existing Hiera users, but I hope to solicit good feedback from those
not using Hiera as well.

The problem, as it currently exists, is that Puppet (core) has no
good first-class mechanism for separating configuration data from
manifests. Everything from $faked_namespaces__in__variable_names to
specialty “params” classes have been tried, with varying degrees of
success.

Hiera came along as another solution to this problem, and provided a
useful abstraction for hierarchical data lookup, but life as a
plugin meant that certain integrations were still difficult. Our aim
in Puppet 3.0 is to tighten up these integrations (making Hiera more
useful), provide first-class data separation solution (for those not
already using Hiera), and to provide a simple and safe migration for
those currently using Hiera.

Here’s what’s new:

* Hiera data keys can be namespaced


* (e.g. 'dns::nameserver': '8.8.8.8' )
* Namespaced data will automatically populate class parameters

* (e.g. 'dns::nameserver' will be automatically looked up for
the $nameserver parameter when you include dns )


Here’s what’s changed:

* The hiera-puppet module will no longer be required


* It should, however, still continue to work
* The heira() function (from the hiera-puppet module) will be
superceded by the lookup() function (in Puppet core)
* hiera_include() will not be ported; include now properly accept
arrays, making this redundant


Here’s what were still wondering about:

* How should we integrate hiera_array() and hiera_hash() ?
* How should we integrate hiera ’s “default” and “override”
parameters?
* How should we handle overlaps between data supplied by Hiera
and data supplied in a parameterized class include?


If you’re interested in test-driving the new functionality for
yourself, checkout out the master branch of the Puppet repository on
Github , install Hiera (with gem install hiera ) and make the
following configuration changes:

* Set data_binding_terminus to hiera
* If you have a Hiera configuration file, set hiera_config to
point to it.
* If you don’t, create a file in ${confdir}/hiera.yaml with these
contents:


:backend:

- yaml

:yaml:

:datadir: $confdir/data

:hierarchy:

- %{certname}

- %{environment}

- global

This sets up a default hierarchy that looks for values in
${confdir}/data , first in the ${certname}.yml file, then in the
${environment}.yml file, then in global.yml .

That should be it! Please, let us know if you're having trouble
getting started, or if you have questions, concerns, thoughts, or
opinions about any of this.

Thanks!


--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/GxVliXSX5u0J .
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

---
hiera_var:
user:
value: 'my_value'
*module/params.pp*
$hiera_var = hiera_hash('hiera_var')
$sorted_var = sort(keys($hiera_var))
*module/template/sometemplate.pp*
<% scope.lookupvar('module::params::sorted_var').each do |item| -%><%
scope.lookupvar('module::params::hiera_var')[item].each do |key,value| -%>
<%= user %> <%= key %> <%= value %>
<% end -%><% end -%>
On Thursday, May 3, 2012 10:05:14 AM UTC-7, Pieter van de Bruggen wrote:

Greetings,

As many of you may be aware, Hiera will be tightly integrated into Puppet
in the upcoming release of Puppet 3.0. As a final sanity check of this
work, I’d like to open our current plan for integration up for feedback.
This is particularly for feedback from existing Hiera users, but I hope to
solicit good feedback from those not using Hiera as well.

The problem, as it currently exists, is that Puppet (core) has no good
first-class mechanism for separating configuration data from manifests.
Everything from $faked_namespaces__in__variable_names to specialty
“params” classes have been tried, with varying degrees of success.

Hiera came along as another solution to this problem, and provided a
useful abstraction for hierarchical data lookup, but life as a plugin meant
that certain integrations were still difficult. Our aim in Puppet 3.0 is to
tighten up these integrations (making Hiera more useful), provide
first-class data separation solution (for those not already using Hiera),
and to provide a simple and safe migration for those currently using Hiera.

Here’s what’s new:

- Hiera data keys can be namespaced
- (e.g. 'dns::nameserver': '8.8.8.8')
- Namespaced data will automatically populate class parameters
- (e.g. 'dns::nameserver' will be automatically looked up for the
$nameserver parameter when you include dns)

Here’s what’s changed:

- The hiera-puppet module will no longer be required
- It should, however, still continue to work
- The heira() function (from the hiera-puppet module) will be
superceded by the lookup() function (in Puppet core)
- hiera_include() will not be ported; include now properly accept
arrays, making this redundant

Here’s what were still wondering about:

- How should we integrate hiera_array() and hiera_hash()?
- How should we integrate hiera’s “default” and “override” parameters?
- How should we handle overlaps between data supplied by Hiera and
data supplied in a parameterized class include?

If you’re interested in test-driving the new functionality for yourself,
checkout out the master branch of the Puppet repository on Github<https://github.com/puppetlabs/puppet>,
install Hiera (with gem install hiera) and make the following
configuration changes:

- Set data_binding_terminus to hiera
- If you have a Hiera configuration file, set hiera_config to point to
it.
- If you don’t, create a file in ${confdir}/hiera.yaml with these
contents:

:backend:

- yaml

:yaml:

:datadir: $confdir/data

:hierarchy:

- %{certname}

- %{environment}

- global

This sets up a default hierarchy that looks for values in ${confdir}/data,
first in the ${certname}.yml file, then in the ${environment}.yml file,
then in global.yml.

That should be it! Please, let us know if you're having trouble getting
started, or if you have questions, concerns, thoughts, or opinions about
any of this.

Thanks!

On 5 May 2012, at 17:26, Trevor Vaughan wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With the integration of Hiera, is the extlookup functionality being deprecated?

If so, I would like to ask that the team add a Hiera backend for the existing extlookup data sources.

I know that it won't support hashes but it would help to break the view that Puppet routinely breaks existing functionality on upgrades.

Thanks,

Trevor
- --
Trevor Vaughan
Vice President, Onyx Point, Inc.
email: [email protected]
phone: 410-541-ONYX (6699)
pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to [email protected].
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

<tvaughan.vcf>

On Sat, May 5, 2012 at 9:26 AM, Trevor Vaughan wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

With the integration of Hiera, is the extlookup functionality being
deprecated?

If so, I would like to ask that the team add a Hiera backend for the
existing extlookup data sources.

I know that it won't support hashes but it would help to break the view
that Puppet routinely breaks existing functionality on upgrades.

On May 3, 2012, at 1:05 PM, Pieter van de Bruggen wrote:
As many of you may be aware, Hiera will be tightly integrated into Puppet in the upcoming release of Puppet 3.0.

On Mon, May 7, 2012 at 7:22 AM, Derek J. Balling wrote:
On May 3, 2012, at 1:05 PM, Pieter van de Bruggen wrote:

As many of you may be aware, Hiera will be tightly integrated into Puppet
in the upcoming release of Puppet 3.0.

What will this mean for sites that have specifically avoided using Hiera?
I would presume that this should just mean "business as usual", with no
significant impact?

On May 7, 2012, at 11:54 AM, Nigel Kersten wrote:
Yes. If you're not planning to externalize data at all, the only difference you'll see is that you'll now have Hiera installed on your system when you install Puppet 3.0.

On Mon, May 07, 2012 at 12:26:32PM -0400, Derek J. Balling wrote:
Cool. Was really panicking there for a moment.

I kinda like having "human readable" manifests as opposed to "machine readable" YAML :-)

Cheers,
D


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to [email protected].
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

On May 7, 2012, at 12:47 PM, Christopher Wood wrote:
Wrapper script (similar concept for anywhere with a yaml reader):
Still have to *write* the YAML files.
Out of nosiness, how do you handle the edgier cases in your manifests? For example, how everything is a standard syslog client, except for this host which is loghost, and these couple of clients which all do different things with syslog?

Still have to *write* the YAML files.

Oh yeah. This is my big motivator sticking with extlookup and not
converting to Hiera - anyone in the team can do CSV. But YAML is
starting to push well into the dev camp and far away from ops (IMHO)

----- Original Message -----
From: "John Warburton" <[email protected]>
To: [email protected]
Sent: Tuesday, May 8, 2012 12:14:48 PM
Subject: Re: [Puppet Users] Puppet 3.0 and Hiera


On 8 May 2012 20:00, Derek J. Balling wrote:



Still have to *write* the YAML files.



Oh yeah. This is my big motivator sticking with extlookup and not
converting to Hiera - anyone in the team can do CSV. But YAML is
starting to push well into the dev camp and far away from ops (IMHO)

On Tue, May 08, 2012 at 06:00:07AM -0400, Derek J. Balling wrote:
Still have to *write* the YAML files.

We would have the syslog module have all the "common" files/packages/services/etc. declared, and then simply do something like:

if ($fqdn =~ /^syslog/)
{
..... log host specific stuff .....
}
else
{
..... NON-log host specific stuff .....
}

doesn't seem like rocket science, and makes it clear, all in one place, what's "common", and what's "special".

Puppet provides some wonderful decision-making capabilities in the manifests themselves, I've yet to see any need to have some ugly machine-readable formatted file to solve a problem.

But I *have* seen the parts of our organization that DO use Hiera complaining loudly that they can't easily figure out which of a slurry of YAML files is changing some setting/variable to a value they're not expecting.

Cheers,
D

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to [email protected].
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

On Thu, May 3, 2012 at 10:05 AM, Pieter van de Bruggen wrote:

Greetings,

As many of you may be aware, Hiera will be tightly integrated into Puppet
in the upcoming release of Puppet 3.0. As a final sanity check of this
work, I’d like to open our current plan for integration up for feedback.
This is particularly for feedback from existing Hiera users, but I hope to
solicit good feedback from those not using Hiera as well.

The problem, as it currently exists, is that Puppet (core) has no good
first-class mechanism for separating configuration data from manifests.
Everything from $faked_namespaces__in__variable_names to specialty
“params” classes have been tried, with varying degrees of success.

Hiera came along as another solution to this problem, and provided a
useful abstraction for hierarchical data lookup, but life as a plugin meant
that certain integrations were still difficult. Our aim in Puppet 3.0 is to
tighten up these integrations (making Hiera more useful), provide
first-class data separation solution (for those not already using Hiera),
and to provide a simple and safe migration for those currently using Hiera.

Here’s what’s new:

- Hiera data keys can be namespaced
- (e.g. 'dns::nameserver': '8.8.8.8')
- Namespaced data will automatically populate class parameters
- (e.g. 'dns::nameserver' will be automatically looked up for the
$nameserver parameter when you include dns)

Here’s what’s changed:

- The hiera-puppet module will no longer be required
- It should, however, still continue to work
- The heira() function (from the hiera-puppet module) will be
superceded by the lookup() function (in Puppet core)
- hiera_include() will not be ported; include now properly accept
arrays, making this redundant

Here’s what were still wondering about:

- How should we integrate hiera_array() and hiera_hash()?
- How should we integrate hiera’s “default” and “override” parameters?
- How should we handle overlaps between data supplied by Hiera and
data supplied in a parameterized class include?

If you’re interested in test-driving the new functionality for yourself,
checkout out the master branch of the Puppet repository on Github<https://github.com/puppetlabs/puppet>,
install Hiera (with gem install hiera) and make the following
configuration changes:

- Set data_binding_terminus to hiera
- If you have a Hiera configuration file, set hiera_config to point to
it.
- If you don’t, create a file in ${confdir}/hiera.yaml with these
contents:

:backend:

- yaml

:yaml:

:datadir: $confdir/data

:hierarchy:

- %{certname}

- %{environment}

- global

This sets up a default hierarchy that looks for values in ${confdir}/data,
first in the ${certname}.yml file, then in the ${environment}.yml file,
then in global.yml.

That should be it! Please, let us know if you're having trouble getting
started, or if you have questions, concerns, thoughts, or opinions about
any of this.

Thanks!