----- Original Message -----
From: "Eric Sorenson" <email@example.com
Sent: Tuesday, October 22, 2013 2:14:59 AM
Subject: Re: [Puppet Users] Re: Status of Data in modules
Another round of thanks for the replies to this thread. I apologize that
almost as soon as I posted it, I got pulled off onto another project and
wasn't able to follow up until now. Replies inline below, and there are
probably a couple more coming to different branches (damn I miss Usenet
John Bollinger wrote:
We agree on most of what you said, but it doesn't seem very responsive to
the comments to which they ostensibly reply. I am in no way arguing
against the idea of the data in modules subsystem. It is a fantastic idea,
and long past due. I *am* concerned, however, about the new approach Eric
proposed. I suggested a more general approach than (my understanding of)
the one he described, one not tied specifically to ::params classes.
Inasmuch as you disfavor ::params classes, I would think that you would
find much to like about my counterproposal. Indeed, I think my proposal is
very much like the original prototype you floated.
John I didn't see a more detailed description of what you're proposing; is
this section (quoted from upthread) what you're referring to?
Do I understand correctly that you set out to get rid of the ::params class
pattern, but now you favor an approach that depends on that pattern?
Heh, well when you put it that way...
Why is that better than being more general: enable an implicit
lowest-priority hierarchy level for values of form 'modulename::variable',
drawing on data from per-module data files such as
If I understand this correctly this is slightly different (and probably
inadequate from RI's standpoint), because it just adds another 'category'
(in the ARM-9 sense) to the end of each lookup, and what RI and others
propose is to have another _complete hiera invocation_ inside the module
owning a class parameter's namespace the end of each unsuccessful
site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
defined, and a tree of data files. (params.pp does this by letting
old-school puppet DSL logic determine your "hierarchy")
I also talked to a user today who wants data from modules (by doing hash
key merge on a parameter's class::subclass::varname) from *any* module in
the modulepath to contribute, say, sudoers rules to the sudo module from
other site-written modules that require particular sudoers stanzas. So I'm
trying to consider how to pull all of this together without making a O(n^n)
Your comments are good and addressed in later replies, especially related
data mangling. This is a common problem in all languages, data almost
arrives in the final form and all programming languages have patterns for
retrieving data, validating and mangling it. We just need to introduce
This is really interesting, and not something that's come up so far
AFAIK. It ties in somewhat to https://projects.puppetlabs.com/issues/20199
, needing a way to indicate the data type of something that's looked up
implicitly with data bindings, but introduces another layer around
retrieving and modifying data as it flows back towards puppet, which I
hadn't considered. That is what the "code-in-data" people are asking for,
that ended up with
arbitrary puppet functions inside hiera curly brace expansion. Would love
thoughts on how to do that in a generally useful, lightweight way.
I, obviously, share your concern with the current round of proposals.
in module querying only params.pp is literally the worst possible
one can make in this regard. It would be a massive step backward. Might
well just go ahead and unmerge hiera if the goal is to not learn anything
its design and incredibly wide adoption.
Oh surely there's way worse suggestions out there :)
I do think it is a mistake to focus on eliminating all need for ::params
classes as a goal of the initiative, however. Likely *most* need for them
can be redirected to a relatively simple data-in-modules subsystem, and
that would be well, but the initiative does not fail if some need for the
::params class pattern remains.
yeah, as per the other replies - eliminate *storing data* in params.pp but
validate/mangle in something like params.pp. That is in the event that
no-one delivers a layer of data validation around data bindings and hiera.
So it doesn't seem helpful to get data-bindings integration via puppet
code, even as a first step?
No, I don't think it's much more work to get to a better place - as is evident
from a PR I sent that does this in about 10% of the code of whats shipped in
Given appropriate data in modules the puppetlabs/ntp module removes the params.pp
entirely and becomes just this: http://p.devco.net/437/
combined with a number
of json files - one per OS.
The key point here is that the data layer provides just the data and that - as
now - the DSL layer provides validation and mangling. Not much mangling in
this example but you can see how here you'd derive data and everything would
just access $ntp::foo
This is a massive simplification and improvement over having this duplication:https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/init.pp#L2-21
and well the whole of params.pp really.
It's clear here that there's no big hurdle in taking pure data validating it and
deriving new data from it, we already have a DSL thats reasonably ok at this
and the new parser makes it even better at that task - though not being able to
reassign variables in the scope of a class still suck.
In the case where you'd like the validation/mangling in a different place you
can move that to a class that is delegated for this purpose, this example is
the first thing I tried, it's not great but enough to show the pattern one
might come up with... http://p.devco.net/438
This receives the external data into the ntp class and then has a class that
takes that data, validates and mangles it and exposes the resulting data as
$ntp::model::foo. It's similar in spirit as the params.pp is today, provides
validation etc separate from the data and achieves the same end goal as embedding
logic in the data without doing that. Errors are richer and clearer the complexity
of what data you can derive is greater and overall it's just superior to trying
to embed even this simple bit of logic into the data. This is valid for both this
weird model class and the first example ( http://p.devco.net/437/
This shows validation and data mangling that exceeds what could comfortably
be done with code inside the data. And means existing approaches wrt unit tests
etc remains valid. You did plan to come up with a unit test framework, linter,
parser validator etc for your data right since now it would contain logic? I'd
love to see how these would look and I hope the answer to the code-in-data crowd
is not that they don't think its needed.
So to be clear when I said patterns for validation and mangling of data needs
to be come up with I do not say we need to invent new things in puppet, just
use puppet. Eventually sure we might add some schema to the data but as a
first iteration on data in modules I think this is a suitable start point and
would naturally progress towards data schema etc.
A bit more on the code in data model that people are saying is indispensable.
You're basically saying you could never use a database that did not have stored
procedures. The stored procedure approach has been discredited far and wide, I
won't get into it, google it many people have written much on the subject and
if you look at any of the big frameworks today stored procs just don't feature.
In the case of Puppet they would trap your data into a form only parsable by
Puppet. Hiera has always had the ability to be used as a gem to use your
data outside of puppet. The pluggable backend has been a major win in it's
adoption because people want outside data in Puppet (something we're losing in
the current round of suggestions). Suggesting the only usable data would be
ones that only Puppet could parse is quite short sighted.
I definitely agree hiera data in general needs a way to do validation, but
the semantics you described of "requiring an integer between 10 and 20"
would be additional complexity on top of Henrik's type system. (That work
correct, thats why I think bringing in the type system as part of this is a
was foundational BTW, not specific to the data-in-modules binder as you
said up-thread, so it can be reused independently of ARM-9)
this has such wide implications that I do not believe enough work has been
done to see how a type system would effect the rest of puppet, so bringing
it in as part of this work is not the right time. It's a overcomplex distraction
when what puppet desperately need is to focus on the data problem and stop
punting on it and stop over engineering and trying to solve 100 problems in
one go, it's not needed. The PR I sent does this in a very focussed manner
in line with current design and future proof way.
It's no secret the future parser is far from ready for general use and the
current trajectory you're taking with data in modules is to postpone their
usefulness even further into the future till a time where the future parser
is usable, performant and bug free. Meanwhile every user of Puppet is being
held back by a module system that lacks a clear data system.