FAQ

On Friday, October 11, 2013 1:09:23 PM UTC-5, Eric Sorenson wrote:

Thanks to everyone who kicked the tires on the experimental data in
modules feature included in Puppet 3.3.0. We got a lot of feedback, some
cool proof-of-concept modules, and a definitive conclusion to the
experiment.

The idea of including a module-specific hiera backend is centered around
one primary use case: replacing the 'params class pattern', a common idiom
in Puppet modules that's described in the [Using Parameterized
Classes][param-classes] guide.


I guess I wasn't following this closely enough to realize that getting rid
of the "params" class pattern was an objective. I thought this was a
somewhat more general initiative.

[...]

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend
for a value of 'classname::variable' in the file
'modules/classname/manifests/params.pp', which simplifies class definition
and provides consistency with other hiera backends. As a module author,
you'd still leave your logic for variables in params.pp, but they'd be
implicitly looked up via data bindings as the class is declared, after
consulting site-wide hiera.

Do I understand correctly that you set out to get rid of the ::params class
pattern, but now you favor an approach that depends on that pattern? Why
is that better than being more general: enable an implicit lowest-priority
hierarchy level for values of form 'modulename::variable', drawing on data
from per-module data files such as modules/modulename/data.yaml?


John

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • David Schmitt at Oct 15, 2013 at 5:52 am

    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends. As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.


    Regards, David

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jcbollinger at Oct 15, 2013 at 1:08 pm

    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends. As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 15, 2013 at 1:35 pm

    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet-users@googlegroups.com
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends. As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and it's
    one file that tends to include data for many different roles/sources/uses.

    You have to consider the main reasons for separating data from code in order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:

        https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

    No-one would call that maintainable or readable vs say having AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

      * Adding FooOS support will not break existing supported OS support.
        FooOS.json is only going to be read on FooOS machines.
      * They do not have to worry about complex merge conflicts on busy modules
        such as the ones you'd find internal to large companies vs many team
        members editing a single param.pp
      * There's no syntax and stuff to bother about, its pretty easy evaluate
        the data and to pre/post commit check this stuff. Contributor dont have
        to test extensively to ensure he didnt accidentally mess up params.pp
        complex nested statements in some subtle manner
      * In large environments if you have strict change control etc, the previous
        points help things a lot, you can easily reason about the implications and you
        can be sure they wont affect existing systems. It's just data that will affect
        a small subset users.

    This improves the maintainers life because:

      * They can find it easier to merge new OS support because the change is contained
        in seperate files and easy to evaluate
      * Fewer complex merge commits and easier, cleaner, commit history
      * The code is simpler and generally easier to maintain in the long term

    This improves the module user life because:

      * He can just look at existing data files and know without having to parse complex
        nested case statements what the available overridable data is and what the keys
        would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature behave consistently
    and predictably with the current more or less universally accepted standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend has been
    broken for ages and not been missed because it does not provide a solution that solves
    the above points. And so the data separation around params.pp will not solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes code again with
    all the related problems.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Chuck at Oct 15, 2013 at 5:13 pm
    I totally agree with R.I. on this.
    On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:


    ----- Original Message -----
    From:
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's
    one file that tends to include data for many different roles/sources/uses.

    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

    No-one would call that maintainable or readable vs say having AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

    * Adding FooOS support will not break existing supported OS support.
    FooOS.json is only going to be read on FooOS machines.
    * They do not have to worry about complex merge conflicts on busy modules
    such as the ones you'd find internal to large companies vs many team
    members editing a single param.pp
    * There's no syntax and stuff to bother about, its pretty easy evaluate
    the data and to pre/post commit check this stuff. Contributor dont
    have
    to test extensively to ensure he didnt accidentally mess up params.pp
    complex nested statements in some subtle manner
    * In large environments if you have strict change control etc, the
    previous
    points help things a lot, you can easily reason about the implications
    and you
    can be sure they wont affect existing systems. It's just data that
    will affect
    a small subset users.

    This improves the maintainers life because:

    * They can find it easier to merge new OS support because the change is
    contained
    in seperate files and easy to evaluate
    * Fewer complex merge commits and easier, cleaner, commit history
    * The code is simpler and generally easier to maintain in the long term

    This improves the module user life because:

    * He can just look at existing data files and know without having to
    parse complex
    nested case statements what the available overridable data is and what
    the keys
    would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend
    has been
    broken for ages and not been missed because it does not provide a solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes code
    again with
    all the related problems.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Alessandro Franceschi at Oct 16, 2013 at 9:54 am

    On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:

    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's
    one file that tends to include data for many different roles/sources/uses.

    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

    No-one would call that maintainable or readable vs say having AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

    * Adding FooOS support will not break existing supported OS support.
    FooOS.json is only going to be read on FooOS machines.
    * They do not have to worry about complex merge conflicts on busy modules
    such as the ones you'd find internal to large companies vs many team
    members editing a single param.pp
    * There's no syntax and stuff to bother about, its pretty easy evaluate
    the data and to pre/post commit check this stuff. Contributor dont
    have
    to test extensively to ensure he didnt accidentally mess up params.pp
    complex nested statements in some subtle manner
    * In large environments if you have strict change control etc, the
    previous
    points help things a lot, you can easily reason about the implications
    and you
    can be sure they wont affect existing systems. It's just data that
    will affect
    a small subset users.

    This improves the maintainers life because:

    * They can find it easier to merge new OS support because the change is
    contained
    in seperate files and easy to evaluate
    * Fewer complex merge commits and easier, cleaner, commit history
    * The code is simpler and generally easier to maintain in the long term

    This improves the module user life because:

    * He can just look at existing data files and know without having to
    parse complex
    nested case statements what the available overridable data is and what
    the keys
    would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend
    has been
    broken for ages and not been missed because it does not provide a solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes code
    again with
    all the related problems.
    It's difficult to disagree with such statements, and actually I do agree
    with all of them.

    Just would like to point out pair of notes, not necessarily in
    contradiction with what you wrote:
    - The params pattern was probably the best choice up to now, data in
    modules probably provides an alternative and conceptually better way to do
    the same things, which will also the have practical benefits you well
    pointed out, but, to my understanding, it has some issues that have to be
    addressed:
    -- Is more difficult to write modules, especially in some specific cases,
    as Eric said, we can cope with that but that's worth considering, because
    complexity is never a welcomed word when talking about code.
    -- It's not clear , at least to me, if some real use cases are covered,
    such as the ones where some class parameters or internal vars change
    according to the value provided to other parameters.
    The install case I think is quite clear to express this, and I still
    haven't understood if it's possible to have in the module's hiera.yaml
    something like this:

    ---
    version: 3
    hierarchy:
       - category: 'osfamily'
       - category: 'operatingsystem'

       - category: '^{install}' # Is this possible? Is this the correct syntax?

       - category: 'environment'
       - category: 'common'
           paths:
             - 'is_virtual/${is_virtual}'
             - 'common'


    -- If a case like the above can't be expressed as "pure" hiera data, we'll
    have to turn back in having some data in code (incidentally I don't
    consider it a mortal sin, as we have been doing this in modules all the
    time up to now), either reverting to hiera_puppet or with the usual code
    gymnics with selectors, cases and ifs.

    my2 alyays naive c
    al

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 16, 2013 at 10:14 am

    ----- Original Message -----
    From: "Alessandro Franceschi" <al@lab42.it>
    To: puppet-users@googlegroups.com
    Sent: Wednesday, October 16, 2013 10:54:14 AM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:


    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's
    one file that tends to include data for many different roles/sources/uses.

    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

    No-one would call that maintainable or readable vs say having AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

    * Adding FooOS support will not break existing supported OS support.
    FooOS.json is only going to be read on FooOS machines.
    * They do not have to worry about complex merge conflicts on busy modules
    such as the ones you'd find internal to large companies vs many team
    members editing a single param.pp
    * There's no syntax and stuff to bother about, its pretty easy evaluate
    the data and to pre/post commit check this stuff. Contributor dont
    have
    to test extensively to ensure he didnt accidentally mess up params.pp
    complex nested statements in some subtle manner
    * In large environments if you have strict change control etc, the
    previous
    points help things a lot, you can easily reason about the implications
    and you
    can be sure they wont affect existing systems. It's just data that
    will affect
    a small subset users.

    This improves the maintainers life because:

    * They can find it easier to merge new OS support because the change is
    contained
    in seperate files and easy to evaluate
    * Fewer complex merge commits and easier, cleaner, commit history
    * The code is simpler and generally easier to maintain in the long term

    This improves the module user life because:

    * He can just look at existing data files and know without having to
    parse complex
    nested case statements what the available overridable data is and what
    the keys
    would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend
    has been
    broken for ages and not been missed because it does not provide a solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes code
    again with
    all the related problems.
    It's difficult to disagree with such statements, and actually I do agree
    with all of them.

    Just would like to point out pair of notes, not necessarily in
    contradiction with what you wrote:
    - The params pattern was probably the best choice up to now, data in
    modules probably provides an alternative and conceptually better way to do
    the same things, which will also the have practical benefits you well
    pointed out, but, to my understanding, it has some issues that have to be
    addressed:
    indeed, params.pp is fine in the absence of something better. The goal has to
    be to separate data from code though. To ask why data in params.pp is a problem
    or to ask why it's the right approach is simply to ask why hiera exists at all.
    -- Is more difficult to write modules, especially in some specific cases,
    as Eric said, we can cope with that but that's worth considering, because
    complexity is never a welcomed word when talking about code.
    indeed - in cases where you have to associate some deriving logic with some
    data item the temptation is there to put that logic in the data. This is a
    mistake.

    There's nothing wrong with a hybrid model where you have data - pure data -
    in a data file and then a class similar in spirit to params.pp to take that data
    and massage it and create derived data.

    The missing thing here and probably the elephant in the room is validation
    of that data. The proposed type implementation in the current thing thats
    in 3.3.x is a mistake.

    It's a trojan horse to get some half baked type system into Puppet via the
    back door, it's at odds with everything else in Puppet and simply not the right
    way to go about it, additional complexity that doesn't seem to have any place
    in Puppet. As a means of providing data validation it's very naive - saying
    data should be of type Integer is not enough of a validation.

    We should rather approach this in a way that there be some descriptive language
    that describes the data - ie. the foo::bar key has to be a Integer between 10
    and 20 - this should be something a module author provides and that any data
    be it from paramaterised classes, data bindings or otherwise are subject to
    this validation.
    -- It's not clear , at least to me, if some real use cases are covered,
    such as the ones where some class parameters or internal vars change
    according to the value provided to other parameters.
    The install case I think is quite clear to express this, and I still
    haven't understood if it's possible to have in the module's hiera.yaml
    something like this:

    ---
    version: 3
    hierarchy:
    - category: 'osfamily'
    - category: 'operatingsystem'

    - category: '^{install}' # Is this possible? Is this the correct syntax?
    hiera does support interpolation of data yes - I think that's what you mean?
    not sure where install comes from, I am guessing its a in-scope variable?

    - category: 'environment'
    - category: 'common'
    paths:
    - 'is_virtual/${is_virtual}'
    - 'common'


    -- If a case like the above can't be expressed as "pure" hiera data, we'll
    have to turn back in having some data in code (incidentally I don't
    consider it a mortal sin, as we have been doing this in modules all the
    time up to now), either reverting to hiera_puppet or with the usual code
    gymnics with selectors, cases and ifs.

    It's a critical feature thats always been there - If I understand what you
    mean.


    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 16, 2013 at 10:15 am

    ----- Original Message -----
    From: "R.I.Pienaar" <rip@devco.net>
    To: puppet-users@googlegroups.com
    Sent: Wednesday, October 16, 2013 11:14:12 AM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    ----- Original Message -----
    From: "Alessandro Franceschi" <al@lab42.it>
    To: puppet-users@googlegroups.com
    Sent: Wednesday, October 16, 2013 10:54:14 AM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:


    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic
    that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give
    users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's
    one file that tends to include data for many different
    roles/sources/uses.

    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

    No-one would call that maintainable or readable vs say having AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are
    supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

    * Adding FooOS support will not break existing supported OS support.
    FooOS.json is only going to be read on FooOS machines.
    * They do not have to worry about complex merge conflicts on busy
    modules
    such as the ones you'd find internal to large companies vs many team
    members editing a single param.pp
    * There's no syntax and stuff to bother about, its pretty easy evaluate
    the data and to pre/post commit check this stuff. Contributor dont
    have
    to test extensively to ensure he didnt accidentally mess up params.pp
    complex nested statements in some subtle manner
    * In large environments if you have strict change control etc, the
    previous
    points help things a lot, you can easily reason about the implications
    and you
    can be sure they wont affect existing systems. It's just data that
    will affect
    a small subset users.

    This improves the maintainers life because:

    * They can find it easier to merge new OS support because the change is
    contained
    in seperate files and easy to evaluate
    * Fewer complex merge commits and easier, cleaner, commit history
    * The code is simpler and generally easier to maintain in the long term

    This improves the module user life because:

    * He can just look at existing data files and know without having to
    parse complex
    nested case statements what the available overridable data is and what
    the keys
    would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet
    backend
    has been
    broken for ages and not been missed because it does not provide a
    solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes code
    again with
    all the related problems.
    It's difficult to disagree with such statements, and actually I do agree
    with all of them.

    Just would like to point out pair of notes, not necessarily in
    contradiction with what you wrote:
    - The params pattern was probably the best choice up to now, data in
    modules probably provides an alternative and conceptually better way to do
    the same things, which will also the have practical benefits you well
    pointed out, but, to my understanding, it has some issues that have to be
    addressed:
    indeed, params.pp is fine in the absence of something better. The goal has
    to
    be to separate data from code though. To ask why data in params.pp is a
    problem
    or to ask why it's the right approach is simply to ask why hiera exists at
    all.
    -- Is more difficult to write modules, especially in some specific cases,
    as Eric said, we can cope with that but that's worth considering, because
    complexity is never a welcomed word when talking about code.
    indeed - in cases where you have to associate some deriving logic with some
    data item the temptation is there to put that logic in the data. This is a
    mistake.

    There's nothing wrong with a hybrid model where you have data - pure data -
    in a data file and then a class similar in spirit to params.pp to take that
    data
    and massage it and create derived data.

    The missing thing here and probably the elephant in the room is validation
    of that data. The proposed type implementation in the current thing thats
    in 3.3.x is a mistake.

    It's a trojan horse to get some half baked type system into Puppet via the
    back door, it's at odds with everything else in Puppet and simply not the
    right
    way to go about it, additional complexity that doesn't seem to have any place
    in Puppet. As a means of providing data validation it's very naive - saying
    data should be of type Integer is not enough of a validation.

    We should rather approach this in a way that there be some descriptive
    language
    that describes the data - ie. the foo::bar key has to be a Integer between 10
    and 20 - this should be something a module author provides and that any data
    be it from paramaterised classes, data bindings or otherwise are subject to
    this validation.
    -- It's not clear , at least to me, if some real use cases are covered,
    such as the ones where some class parameters or internal vars change
    according to the value provided to other parameters.
    The install case I think is quite clear to express this, and I still
    haven't understood if it's possible to have in the module's hiera.yaml
    something like this:

    ---
    version: 3
    hierarchy:
    - category: 'osfamily'
    - category: 'operatingsystem'

    - category: '^{install}' # Is this possible? Is this the correct syntax?
    hiera does support interpolation of data yes - I think that's what you mean?
    not sure where install comes from, I am guessing its a in-scope variable?

    - category: 'environment'
    - category: 'common'
    paths:
    - 'is_virtual/${is_virtual}'
    - 'common'


    -- If a case like the above can't be expressed as "pure" hiera data, we'll
    have to turn back in having some data in code (incidentally I don't
    consider it a mortal sin, as we have been doing this in modules all the
    time up to now), either reverting to hiera_puppet or with the usual code
    gymnics with selectors, cases and ifs.

    It's a critical feature thats always been there - If I understand what you
    mean.
    Sorry - it's critical but also missing in the proposed data in module implementation
    which is a oversight.



    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Alessandro Franceschi at Oct 16, 2013 at 11:06 am
    Il giorno mercoledì 16 ottobre 2013 12:14:12 UTC+2, R.I. Pienaar ha scritto:

    ----- Original Message -----
    From: "Alessandro Franceschi" <a...@lab42.it <javascript:>>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Wednesday, October 16, 2013 10:54:14 AM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:


    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies
    class
    definition and provides consistency with other hiera
    backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data
    bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on
    that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data
    files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic
    that
    cannot be expressed in .yaml to calculate default values. Making
    the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is
    a
    viable target. In fact, I don't really understand why it was ever
    an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern,
    either.
    A
    more general data-in-modules feature such as I describe would give
    users
    the option to avoid ::params classes in some cases, and I'm inclined
    to
    think that it would be easier to implement, to understand, and to
    use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's
    one file that tends to include data for many different
    roles/sources/uses.
    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:

    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
    No-one would call that maintainable or readable vs say having
    AIX.json,
    Debian.json and so forth.

    For a community member who wants to add support for a new OS this
    simplifies things a LOT. They can see what operating systems are
    supported
    already and they can easily add a new one by dropping a single file.

    This improves the contributor life cycle significantly:

    * Adding FooOS support will not break existing supported OS support.
    FooOS.json is only going to be read on FooOS machines.
    * They do not have to worry about complex merge conflicts on busy
    modules
    such as the ones you'd find internal to large companies vs many
    team
    members editing a single param.pp
    * There's no syntax and stuff to bother about, its pretty easy
    evaluate
    the data and to pre/post commit check this stuff. Contributor dont
    have
    to test extensively to ensure he didnt accidentally mess up
    params.pp
    complex nested statements in some subtle manner
    * In large environments if you have strict change control etc, the
    previous
    points help things a lot, you can easily reason about the
    implications
    and you
    can be sure they wont affect existing systems. It's just data that
    will affect
    a small subset users.

    This improves the maintainers life because:

    * They can find it easier to merge new OS support because the change
    is
    contained
    in seperate files and easy to evaluate
    * Fewer complex merge commits and easier, cleaner, commit history
    * The code is simpler and generally easier to maintain in the long
    term
    This improves the module user life because:

    * He can just look at existing data files and know without having to
    parse complex
    nested case statements what the available overridable data is and
    what
    the keys
    would be etc.

    There are more but these are the basics

    From a Puppet system perspective its important that this feature
    behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet
    backend
    has been
    broken for ages and not been missed because it does not provide a
    solution
    that solves
    the above points. And so the data separation around params.pp will
    not
    solve the real
    problems either.

    Data simply should not be mixed with logic - because then it becomes
    code
    again with
    all the related problems.
    It's difficult to disagree with such statements, and actually I do agree
    with all of them.

    Just would like to point out pair of notes, not necessarily in
    contradiction with what you wrote:
    - The params pattern was probably the best choice up to now, data in
    modules probably provides an alternative and conceptually better way to do
    the same things, which will also the have practical benefits you well
    pointed out, but, to my understanding, it has some issues that have to be
    addressed:
    indeed, params.pp is fine in the absence of something better. The goal
    has to
    be to separate data from code though. To ask why data in params.pp is a
    problem
    or to ask why it's the right approach is simply to ask why hiera exists at
    all.
    -- Is more difficult to write modules, especially in some specific cases,
    as Eric said, we can cope with that but that's worth considering, because
    complexity is never a welcomed word when talking about code.
    indeed - in cases where you have to associate some deriving logic with
    some
    data item the temptation is there to put that logic in the data. This is
    a
    mistake.

    There's nothing wrong with a hybrid model where you have data - pure data
    -
    in a data file and then a class similar in spirit to params.pp to take
    that data
    and massage it and create derived data.

    The missing thing here and probably the elephant in the room is validation
    of that data. The proposed type implementation in the current thing thats
    in 3.3.x is a mistake.

    It's a trojan horse to get some half baked type system into Puppet via the
    back door, it's at odds with everything else in Puppet and simply not the
    right
    way to go about it, additional complexity that doesn't seem to have any
    place
    in Puppet. As a means of providing data validation it's very naive -
    saying
    data should be of type Integer is not enough of a validation.

    We should rather approach this in a way that there be some descriptive
    language
    that describes the data - ie. the foo::bar key has to be a Integer between
    10
    and 20 - this should be something a module author provides and that any
    data
    be it from paramaterised classes, data bindings or otherwise are subject
    to
    this validation.
    -- It's not clear , at least to me, if some real use cases are covered,
    such as the ones where some class parameters or internal vars change
    according to the value provided to other parameters.
    The install case I think is quite clear to express this, and I still
    haven't understood if it's possible to have in the module's hiera.yaml
    something like this:

    ---
    version: 3
    hierarchy:
    - category: 'osfamily'
    - category: 'operatingsystem'

    - category: '^{install}' # Is this possible? Is this the correct
    syntax?

    hiera does support interpolation of data yes - I think that's what you
    mean?
    not sure where install comes from, I am guessing its a in-scope variable?
    Yes, is a parameter of the class it allow's users to decide how to
    install the class/module's application: via package, downloading and
    extracting a tarball from the official site or whatever.
    When used, some module's parameters change (for example the paths of files)
    and therefore the values of these paths are no more the operatingsystem
    specific ones but depend on other variables ( something like
    $install_destination_dir, for example).
    So the issue here is: if $install == 'upstream' (retrieve the software as a
    zip/tarball from the upstream site and not use the OS package) and
    $install_destination_dir == '/opt' , for example, then the paths of the
    configuration files is different.
    To my understanding in order to reproduce this logic in hiera data, we have
    to add a hierarchy level according to the value of $install and in the
    relevant yaml file (or whatever datastore) interpolate
    $install_destination_dir in order to provide the correct $config_file_path .

    Actually my sample hiera.yml could be just :

    ---
    version: 3
    hierarchy:
       - category: 'osfamily'
       - category: 'operatingsystem'
       - category: 'install' # Class' scope variable , what the correct way to
    reference it?
    ...

    so maybe it's not that hard to do.





    - category: 'environment'
    - category: 'common'
    paths:
    - 'is_virtual/${is_virtual}'
    - 'common'


    -- If a case like the above can't be expressed as "pure" hiera data, we'll
    have to turn back in having some data in code (incidentally I don't
    consider it a mortal sin, as we have been doing this in modules all the
    time up to now), either reverting to hiera_puppet or with the usual code
    gymnics with selectors, cases and ifs.

    It's a critical feature thats always been there - If I understand what you
    mean.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jcbollinger at Oct 16, 2013 at 3:23 pm

    On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:

    ----- Original Message -----
    From: "jcbollinger"
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's

    Eric observes -- and this comes as no surprise to me -- that the reason it
    didn't work well to eliminate params.pp was that sometimes you *need* code
    to help set appropriate default parameter values. Or at least that is the
    practical conclusion that arose from field testing the original approach.
    I am all for pushing the actual data out of manifests, but you still need a
    place to hang code. Pushing code out into the data to allow ::params
    classes to be dropped does not achieve the objective of separating data
    from code.


    one file that tends to include data for many different roles/sources/uses.

    That's a fair observation, though a bit usage-oriented. Nothing prevents
    users from creating a separate class::params class for every single class,
    which would at least partially address that. In any case, the proposed new
    data in modules approach does nothing to address that concern -- if
    anything, it magnifies that problem. If the objective is still to avoid
    params.pp, then why does the revised data in modules proposal provide data *
    only* to ::params classes?


    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
    That sort of thing is not really the issue. I think we pretty much all
    agree that pushing that out to an external data source is a win. The issue
    is with cases where the rules for choosing parameter defaults are more
    complex than a simple switch based on one fact. As I understand Eric,
    field testing showed that users wanted to be able to perform essentially
    arbitrary logic to *compute* the data provided to modules. I agree with
    his conclusion that rolling such logic into the data is as bad as putting
    the data into ::params classes. Thus both are needed, and ::params classes
    or an equivalent serve a useful continuing role.


    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend
    has been
    broken for ages and not been missed because it does not provide a solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my proposal is
    very much like the original prototype you floated.

    I do think it is a mistake to focus on eliminating all need for ::params
    classes as a goal of the initiative, however. Likely *most* need for them
    can be redirected to a relatively simple data-in-modules subsystem, and
    that would be well, but the initiative does not fail if some need for the
    ::params class pattern remains.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 16, 2013 at 3:33 pm

    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet-users@googlegroups.com
    Sent: Wednesday, October 16, 2013 4:23:17 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:


    ----- Original Message -----
    From: "jcbollinger"
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 15, 2013 2:08:10 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
    On 14.10.2013 16:16, jcbollinger wrote:
    So the proposed solution at this point is:
    - enable an implicit data-binding lookup against the
    hiera-puppet
    backend for a value of 'classname::variable' in the file
    'modules/classname/manifests/params.pp', which simplifies class
    definition and provides consistency with other hiera backends.
    As a
    module author, you'd still leave your logic for variables in
    params.pp, but they'd be implicitly looked up via data bindings
    as
    the class is declared, after consulting site-wide hiera.



    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern? Why is that better than being more general: enable an
    implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable', drawing on data from per-module data files
    such
    as modules/modulename/data.yaml?
    AIUI, prototyping revealed that many params classes contain logic that
    cannot be expressed in .yaml to calculate default values. Making the
    fallback lookup check the params class, the logic can be preserved
    while
    avoiding verbose boilerplate code like the params_lookup calls in
    Alessandro's modules.
    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.
    there are many reasons to avoid params.pp. It's *code* not *data* and
    it's

    Eric observes -- and this comes as no surprise to me -- that the reason it
    didn't work well to eliminate params.pp was that sometimes you *need* code
    to help set appropriate default parameter values. Or at least that is the
    practical conclusion that arose from field testing the original approach.
    I am all for pushing the actual data out of manifests, but you still need a
    place to hang code. Pushing code out into the data to allow ::params
    classes to be dropped does not achieve the objective of separating data
    from code.


    one file that tends to include data for many different roles/sources/uses.

    That's a fair observation, though a bit usage-oriented. Nothing prevents
    users from creating a separate class::params class for every single class,
    which would at least partially address that. In any case, the proposed new
    data in modules approach does nothing to address that concern -- if
    anything, it magnifies that problem. If the objective is still to avoid
    params.pp, then why does the revised data in modules proposal provide data *
    only* to ::params classes?


    You have to consider the main reasons for separating data from code in
    order
    to understand the motivation.

    When you have a params.pp you end up with stuff like this:


    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
    That sort of thing is not really the issue. I think we pretty much all
    agree that pushing that out to an external data source is a win. The issue
    is with cases where the rules for choosing parameter defaults are more
    complex than a simple switch based on one fact. As I understand Eric,
    field testing showed that users wanted to be able to perform essentially
    arbitrary logic to *compute* the data provided to modules. I agree with
    his conclusion that rolling such logic into the data is as bad as putting
    the data into ::params classes. Thus both are needed, and ::params classes
    or an equivalent serve a useful continuing role.


    From a Puppet system perspective its important that this feature behave
    consistently
    and predictably with the current more or less universally accepted
    standard of data
    seperation - hiera. And hiera is all about data, the hiera puppet backend
    has been
    broken for ages and not been missed because it does not provide a solution
    that solves
    the above points. And so the data separation around params.pp will not
    solve the real
    problems either.
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my proposal is
    very much like the original prototype you floated.
    Your comments are good and addressed in later replies, especially related to
    data mangling. This is a common problem in all languages, data almost never
    arrives in the final form and all programming languages have patterns for
    retrieving data, validating and mangling it. We just need to introduce
    similar patterns.

    I, obviously, share your concern with the current round of proposals. Data
    in module querying only params.pp is literally the worst possible suggestion
    one can make in this regard. It would be a massive step backward. Might as
    well just go ahead and unmerge hiera if the goal is to not learn anything from
    its design and incredibly wide adoption.

    I do think it is a mistake to focus on eliminating all need for ::params
    classes as a goal of the initiative, however. Likely *most* need for them
    can be redirected to a relatively simple data-in-modules subsystem, and
    that would be well, but the initiative does not fail if some need for the
    ::params class pattern remains.
    yeah, as per the other replies - eliminate *storing data* in params.pp but
    validate/mangle in something like params.pp. That is in the event that
    no-one delivers a layer of data validation around data bindings and hiera.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Eric Sorenson at Oct 22, 2013 at 1:15 am
    Another round of thanks for the replies to this thread. I apologize that
    almost as soon as I posted it, I got pulled off onto another project and
    wasn't able to follow up until now. Replies inline below, and there are
    probably a couple more coming to different branches (damn I miss Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're proposing; is
    this section (quoted from upthread) what you're referring to?

    Do I understand correctly that you set out to get rid of the ::params class
    pattern, but now you favor an approach that depends on that pattern?

    Heh, well when you put it that way...

    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form 'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and probably
    inadequate from RI's standpoint), because it just adds another 'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and others
    propose is to have another _complete hiera invocation_ inside the module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")

    I also talked to a user today who wants data from modules (by doing hash
    key merge on a parameter's class::subclass::varname) from *any* module in
    the modulepath to contribute, say, sudoers rules to the sudo module from
    other site-written modules that require particular sudoers stanzas. So I'm
    trying to consider how to pull all of this together without making a O(n^n)
    complexity explosion.

    RI replied:
    Your comments are good and addressed in later replies, especially related
    to
    data mangling. This is a common problem in all languages, data almost
    never
    arrives in the final form and all programming languages have patterns for
    retrieving data, validating and mangling it. We just need to introduce
    similar patterns.
    This is really interesting, and not something that's come up so far
    AFAIK. It ties in somewhat to https://projects.puppetlabs.com/issues/20199
      , needing a way to indicate the data type of something that's looked up
    implicitly with data bindings, but introduces another layer around
    retrieving and modifying data as it flows back towards puppet, which I
    hadn't considered. That is what the "code-in-data" people are asking for,
    like https://github.com/puppetlabs/hiera/pull/152 that ended up with
    arbitrary puppet functions inside hiera curly brace expansion. Would love
    thoughts on how to do that in a generally useful, lightweight way.

    I, obviously, share your concern with the current round of proposals.
    Data
    in module querying only params.pp is literally the worst possible
    suggestion
    one can make in this regard. It would be a massive step backward. Might
    as
    well just go ahead and unmerge hiera if the goal is to not learn anything
    from
    its design and incredibly wide adoption.
    Oh surely there's way worse suggestions out there :)

    I do think it is a mistake to focus on eliminating all need for ::params
    classes as a goal of the initiative, however. Likely *most* need for them
    can be redirected to a relatively simple data-in-modules subsystem, and
    that would be well, but the initiative does not fail if some need for the
    ::params class pattern remains.
    yeah, as per the other replies - eliminate *storing data* in params.pp but
    validate/mangle in something like params.pp. That is in the event that
    no-one delivers a layer of data validation around data bindings and hiera.
    So it doesn't seem helpful to get data-bindings integration via puppet
    code, even as a first step?

    I definitely agree hiera data in general needs a way to do validation, but
    the semantics you described of "requiring an integer between 10 and 20"
    would be additional complexity on top of Henrik's type system. (That work
    was foundational BTW, not specific to the data-in-modules binder as you
    said up-thread, so it can be reused independently of ARM-9)

    --eric0

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 22, 2013 at 8:37 am

    ----- Original Message -----
    From: "Eric Sorenson" <eric.sorenson@puppetlabs.com>
    To: puppet-users@googlegroups.com
    Sent: Tuesday, October 22, 2013 2:14:59 AM
    Subject: Re: [Puppet Users] Re: Status of Data in modules

    Another round of thanks for the replies to this thread. I apologize that
    almost as soon as I posted it, I got pulled off onto another project and
    wasn't able to follow up until now. Replies inline below, and there are
    probably a couple more coming to different branches (damn I miss Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're proposing; is
    this section (quoted from upthread) what you're referring to?

    Do I understand correctly that you set out to get rid of the ::params class
    pattern, but now you favor an approach that depends on that pattern?

    Heh, well when you put it that way...

    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form 'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and probably
    inadequate from RI's standpoint), because it just adds another 'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and others
    propose is to have another _complete hiera invocation_ inside the module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")

    I also talked to a user today who wants data from modules (by doing hash
    key merge on a parameter's class::subclass::varname) from *any* module in
    the modulepath to contribute, say, sudoers rules to the sudo module from
    other site-written modules that require particular sudoers stanzas. So I'm
    trying to consider how to pull all of this together without making a O(n^n)
    complexity explosion.

    RI replied:
    Your comments are good and addressed in later replies, especially related
    to
    data mangling. This is a common problem in all languages, data almost
    never
    arrives in the final form and all programming languages have patterns for
    retrieving data, validating and mangling it. We just need to introduce
    similar patterns.
    This is really interesting, and not something that's come up so far
    AFAIK. It ties in somewhat to https://projects.puppetlabs.com/issues/20199
    , needing a way to indicate the data type of something that's looked up
    implicitly with data bindings, but introduces another layer around
    retrieving and modifying data as it flows back towards puppet, which I
    hadn't considered. That is what the "code-in-data" people are asking for,
    like https://github.com/puppetlabs/hiera/pull/152 that ended up with
    arbitrary puppet functions inside hiera curly brace expansion. Would love
    thoughts on how to do that in a generally useful, lightweight way.

    I, obviously, share your concern with the current round of proposals.
    Data
    in module querying only params.pp is literally the worst possible
    suggestion
    one can make in this regard. It would be a massive step backward. Might
    as
    well just go ahead and unmerge hiera if the goal is to not learn anything
    from
    its design and incredibly wide adoption.
    Oh surely there's way worse suggestions out there :)

    I do think it is a mistake to focus on eliminating all need for ::params
    classes as a goal of the initiative, however. Likely *most* need for them
    can be redirected to a relatively simple data-in-modules subsystem, and
    that would be well, but the initiative does not fail if some need for the
    ::params class pattern remains.
    yeah, as per the other replies - eliminate *storing data* in params.pp but
    validate/mangle in something like params.pp. That is in the event that
    no-one delivers a layer of data validation around data bindings and hiera.
    So it doesn't seem helpful to get data-bindings integration via puppet
    code, even as a first step?
    No, I don't think it's much more work to get to a better place - as is evident
    from a PR I sent that does this in about 10% of the code of whats shipped in
    3.3.0.

    Given appropriate data in modules the puppetlabs/ntp module removes the params.pp
    entirely and becomes just this: http://p.devco.net/437/ combined with a number
    of json files - one per OS.

    The key point here is that the data layer provides just the data and that - as
    now - the DSL layer provides validation and mangling. Not much mangling in
    this example but you can see how here you'd derive data and everything would
    just access $ntp::foo

    This is a massive simplification and improvement over having this duplication:

    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/init.pp#L2-21

    AND

    https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L2-21

    and well the whole of params.pp really.

    It's clear here that there's no big hurdle in taking pure data validating it and
    deriving new data from it, we already have a DSL thats reasonably ok at this
    and the new parser makes it even better at that task - though not being able to
    reassign variables in the scope of a class still suck.

    In the case where you'd like the validation/mangling in a different place you
    can move that to a class that is delegated for this purpose, this example is
    the first thing I tried, it's not great but enough to show the pattern one
    might come up with... http://p.devco.net/438

    This receives the external data into the ntp class and then has a class that
    takes that data, validates and mangles it and exposes the resulting data as
    $ntp::model::foo. It's similar in spirit as the params.pp is today, provides
    validation etc separate from the data and achieves the same end goal as embedding
    logic in the data without doing that. Errors are richer and clearer the complexity
    of what data you can derive is greater and overall it's just superior to trying
    to embed even this simple bit of logic into the data. This is valid for both this
    weird model class and the first example ( http://p.devco.net/437/ )

    This shows validation and data mangling that exceeds what could comfortably
    be done with code inside the data. And means existing approaches wrt unit tests
    etc remains valid. You did plan to come up with a unit test framework, linter,
    parser validator etc for your data right since now it would contain logic? I'd
    love to see how these would look and I hope the answer to the code-in-data crowd
    is not that they don't think its needed.

    So to be clear when I said patterns for validation and mangling of data needs
    to be come up with I do not say we need to invent new things in puppet, just
    use puppet. Eventually sure we might add some schema to the data but as a
    first iteration on data in modules I think this is a suitable start point and
    would naturally progress towards data schema etc.

    A bit more on the code in data model that people are saying is indispensable.
    You're basically saying you could never use a database that did not have stored
    procedures. The stored procedure approach has been discredited far and wide, I
    won't get into it, google it many people have written much on the subject and
    if you look at any of the big frameworks today stored procs just don't feature.
    In the case of Puppet they would trap your data into a form only parsable by
    Puppet. Hiera has always had the ability to be used as a gem to use your
    data outside of puppet. The pluggable backend has been a major win in it's
    adoption because people want outside data in Puppet (something we're losing in
    the current round of suggestions). Suggesting the only usable data would be
    ones that only Puppet could parse is quite short sighted.
    I definitely agree hiera data in general needs a way to do validation, but
    the semantics you described of "requiring an integer between 10 and 20"
    would be additional complexity on top of Henrik's type system. (That work
    correct, thats why I think bringing in the type system as part of this is a
    mistake.
    was foundational BTW, not specific to the data-in-modules binder as you
    said up-thread, so it can be reused independently of ARM-9)
    this has such wide implications that I do not believe enough work has been
    done to see how a type system would effect the rest of puppet, so bringing
    it in as part of this work is not the right time. It's a overcomplex distraction
    when what puppet desperately need is to focus on the data problem and stop
    punting on it and stop over engineering and trying to solve 100 problems in
    one go, it's not needed. The PR I sent does this in a very focussed manner
    in line with current design and future proof way.

    It's no secret the future parser is far from ready for general use and the
    current trajectory you're taking with data in modules is to postpone their
    usefulness even further into the future till a time where the future parser
    is usable, performant and bug free. Meanwhile every user of Puppet is being
    held back by a module system that lacks a clear data system.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jcbollinger at Oct 22, 2013 at 2:13 pm

    On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:
    Another round of thanks for the replies to this thread. I apologize that
    almost as soon as I posted it, I got pulled off onto another project and
    wasn't able to follow up until now. Replies inline below, and there are
    probably a couple more coming to different branches (damn I miss Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my
    proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're proposing; is
    this section (quoted from upthread) what you're referring to?
    Yes.

    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that pattern?

    Heh, well when you put it that way...

    Let's also keep in mind that the purpose of the ::params class pattern is
    not really to serve as a per-module general data repository. Rather, it is
    specifically to provide a means for indirection of class parameter
    defaults. To the extent that ::params classes now do serve as data
    repositories, it is -- or should be -- in service to that purpose, not to a
    broader one. Data in modules is a *complementary*, but more general,
    approach whereby default values expressed in DSL code can in some cases be
    replaced by default values drawn from per-module data. Where data are
    consumed by a module in other ways or for other purposes, there is no
    particular reason why a ::params class should be involved.


    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form 'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and probably
    inadequate from RI's standpoint), because it just adds another 'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and others
    propose is to have another _complete hiera invocation_ inside the module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")
    I don't have any particular objection to implementing data-in-modules as a
    separate full-fledged lookup against a per-module fallback hierarchy, but
    the qualitative differences from what I suggested are subtle. For the most
    part, I think it's just a question of how many levels you can or do add to
    the bottom of the logical hierarchy, whether it's implemented via one call
    to the hiera subsystem or two. There is a difference, however, in the
    behavior of lookups that collected data from across the hierarchy, i.e.
    hiera_hash() and hiera_array(). Those aren't relevant to class parameter
    binding (at this point), but it is worth considering what semantics are
    wanted there, and whether there might be a way for the caller to choose.


    I also talked to a user today who wants data from modules (by doing hash
    key merge on a parameter's class::subclass::varname) from *any* module in
    the modulepath to contribute, say, sudoers rules to the sudo module from
    other site-written modules that require particular sudoers stanzas. So I'm
    trying to consider how to pull all of this together without making a O(n^n)
    complexity explosion.

    I'm with R.I. in suggesting that you get something solid and fundamentally
    sound out soon, even if it doesn't address every user request on the first
    go (or ever). I understand how a confederated data source such as you now
    describe could be useful, but I think such a feature would require a
    significant effort in its own right.

    Furthermore, I think you are fast approaching the point where the data
    subsystem cannot automagically do the right thing in every case. I don't
    think it would be a sin to require some features to be explicitly declared
    or invoked by DSL code. For example, perhaps you want a data access
    function that allows the caller to somehow specify the scope of the data to
    search. Maybe a couple of releases down the line.


    John


    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Oct 22, 2013 at 2:46 pm

    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet-users@googlegroups.com
    Sent: Tuesday, October 22, 2013 3:13:14 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:

    Another round of thanks for the replies to this thread. I apologize that
    almost as soon as I posted it, I got pulled off onto another project and
    wasn't able to follow up until now. Replies inline below, and there are
    probably a couple more coming to different branches (damn I miss Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you would
    find much to like about my counterproposal. Indeed, I think my
    proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're proposing; is
    this section (quoted from upthread) what you're referring to?
    Yes.

    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that pattern?

    Heh, well when you put it that way...

    Let's also keep in mind that the purpose of the ::params class pattern is
    not really to serve as a per-module general data repository. Rather, it is
    specifically to provide a means for indirection of class parameter
    defaults. To the extent that ::params classes now do serve as data
    repositories, it is -- or should be -- in service to that purpose, not to a
    broader one. Data in modules is a *complementary*, but more general,
    approach whereby default values expressed in DSL code can in some cases be
    replaced by default values drawn from per-module data. Where data are
    consumed by a module in other ways or for other purposes, there is no
    particular reason why a ::params class should be involved.


    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form 'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and probably
    inadequate from RI's standpoint), because it just adds another 'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and others
    propose is to have another _complete hiera invocation_ inside the module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")
    I don't have any particular objection to implementing data-in-modules as a
    separate full-fledged lookup against a per-module fallback hierarchy, but
    the qualitative differences from what I suggested are subtle. For the most
    part, I think it's just a question of how many levels you can or do add to
    the bottom of the logical hierarchy, whether it's implemented via one call
    to the hiera subsystem or two. There is a difference, however, in the
    behavior of lookups that collected data from across the hierarchy, i.e.
    hiera_hash() and hiera_array(). Those aren't relevant to class parameter
    binding (at this point), but it is worth considering what semantics are
    wanted there, and whether there might be a way for the caller to choose.


    I also talked to a user today who wants data from modules (by doing hash
    key merge on a parameter's class::subclass::varname) from *any* module in
    the modulepath to contribute, say, sudoers rules to the sudo module from
    other site-written modules that require particular sudoers stanzas. So I'm
    trying to consider how to pull all of this together without making a O(n^n)
    complexity explosion.

    I'm with R.I. in suggesting that you get something solid and fundamentally
    sound out soon, even if it doesn't address every user request on the first
    go (or ever). I understand how a confederated data source such as you now
    describe could be useful, but I think such a feature would require a
    significant effort in its own right.

    Furthermore, I think you are fast approaching the point where the data
    subsystem cannot automagically do the right thing in every case. I don't
    think it would be a sin to require some features to be explicitly declared
    or invoked by DSL code. For example, perhaps you want a data access
    function that allows the caller to somehow specify the scope of the data to
    search. Maybe a couple of releases down the line.
    Absolutely, there will never be a single data solution that solves 100% of problems
    but luckily we have multiple approaches - hiera, ENC, node terminii etc. This user
    can also write his own hiera backend to achieve his goal. Thats the point.

    Aim for a solid fit for the 80% of problems, inform and educate by the solution
    by being prescriptive but with extension points for the 20% of users who have really
    complex problems who can carry the cost of the complexity solving them brings.

    For the rest the design of the tool informs how they approach writing manifests
    and for green fields even how they build infrastructures.

    There is no chance that a 100% solution will be found for the data problem, just
    give up. Sometimes it's ok - as in this use case Eric mentions - to just say no
    that is not in the interest of the larger % of community and we're prioritizing
    shipping something over pleasing 100% of people.

    Just say no, point to extension points. ship something - as long as that something
    isn't impossible to extend for mortals like the arm9 work, because then you have to
    solve 100% of the problems.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Dec 8, 2013 at 10:19 pm

    ----- Original Message -----
    From: "R.I.Pienaar" <rip@devco.net>
    To: puppet-users@googlegroups.com
    Sent: Tuesday, October 22, 2013 3:46:45 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet-users@googlegroups.com
    Sent: Tuesday, October 22, 2013 3:13:14 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:

    Another round of thanks for the replies to this thread. I apologize that
    almost as soon as I posted it, I got pulled off onto another project and
    wasn't able to follow up until now. Replies inline below, and there are
    probably a couple more coming to different branches (damn I miss Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very responsive to
    the comments to which they ostensibly reply. I am in no way arguing
    against the idea of the data in modules subsystem. It is a fantastic idea,
    and long past due. I *am* concerned, however, about the new approach Eric
    proposed. I suggested a more general approach than (my understanding of)
    the one he described, one not tied specifically to ::params classes.
    Inasmuch as you disfavor ::params classes, I would think that you
    would
    find much to like about my counterproposal. Indeed, I think my
    proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're proposing;
    is
    this section (quoted from upthread) what you're referring to?
    Yes.

    Do I understand correctly that you set out to get rid of the ::params
    class pattern, but now you favor an approach that depends on that
    pattern?

    Heh, well when you put it that way...

    Let's also keep in mind that the purpose of the ::params class pattern is
    not really to serve as a per-module general data repository. Rather, it is
    specifically to provide a means for indirection of class parameter
    defaults. To the extent that ::params classes now do serve as data
    repositories, it is -- or should be -- in service to that purpose, not to a
    broader one. Data in modules is a *complementary*, but more general,
    approach whereby default values expressed in DSL code can in some cases be
    replaced by default values drawn from per-module data. Where data are
    consumed by a module in other ways or for other purposes, there is no
    particular reason why a ::params class should be involved.


    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and probably
    inadequate from RI's standpoint), because it just adds another 'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and others
    propose is to have another _complete hiera invocation_ inside the module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")
    I don't have any particular objection to implementing data-in-modules as a
    separate full-fledged lookup against a per-module fallback hierarchy, but
    the qualitative differences from what I suggested are subtle. For the most
    part, I think it's just a question of how many levels you can or do add to
    the bottom of the logical hierarchy, whether it's implemented via one call
    to the hiera subsystem or two. There is a difference, however, in the
    behavior of lookups that collected data from across the hierarchy, i.e.
    hiera_hash() and hiera_array(). Those aren't relevant to class parameter
    binding (at this point), but it is worth considering what semantics are
    wanted there, and whether there might be a way for the caller to choose.


    I also talked to a user today who wants data from modules (by doing hash
    key merge on a parameter's class::subclass::varname) from *any* module in
    the modulepath to contribute, say, sudoers rules to the sudo module from
    other site-written modules that require particular sudoers stanzas. So
    I'm
    trying to consider how to pull all of this together without making a
    O(n^n)
    complexity explosion.

    I'm with R.I. in suggesting that you get something solid and fundamentally
    sound out soon, even if it doesn't address every user request on the first
    go (or ever). I understand how a confederated data source such as you now
    describe could be useful, but I think such a feature would require a
    significant effort in its own right.

    Furthermore, I think you are fast approaching the point where the data
    subsystem cannot automagically do the right thing in every case. I don't
    think it would be a sin to require some features to be explicitly declared
    or invoked by DSL code. For example, perhaps you want a data access
    function that allows the caller to somehow specify the scope of the data to
    search. Maybe a couple of releases down the line.
    Absolutely, there will never be a single data solution that solves 100% of
    problems
    but luckily we have multiple approaches - hiera, ENC, node terminii etc.
    This user
    can also write his own hiera backend to achieve his goal. Thats the point.

    Aim for a solid fit for the 80% of problems, inform and educate by the
    solution
    by being prescriptive but with extension points for the 20% of users who have
    really
    complex problems who can carry the cost of the complexity solving them
    brings.

    For the rest the design of the tool informs how they approach writing
    manifests
    and for green fields even how they build infrastructures.

    There is no chance that a 100% solution will be found for the data problem,
    just
    give up. Sometimes it's ok - as in this use case Eric mentions - to just say
    no
    that is not in the interest of the larger % of community and we're
    prioritizing
    shipping something over pleasing 100% of people.

    Just say no, point to extension points. ship something - as long as that
    something
    isn't impossible to extend for mortals like the arm9 work, because then you
    have to
    solve 100% of the problems.
    I have released a solution to this @ http://www.devco.net/archives/2013/12/08/better-puppet-modules-using-hiera-data.php

    Need as wide as possible feedback and testing, lets move forward.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1288793356.448.1386541177841.JavaMail.zimbra%40devco.net.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Alessandro Franceschi at Dec 8, 2013 at 10:40 pm
    Wow, this looks promising: sane, plain and easy to use.
    Going to test it soon.
    Looking at things in perspective, how do you think this approach will go
    along with the one implemented in Puppet 3.3 ?

    al
    On Sunday, December 8, 2013 11:19:37 PM UTC+1, R.I. Pienaar wrote:


    ----- Original Message -----
    From: "R.I.Pienaar" <r...@devco.net <javascript:>>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 22, 2013 3:46:45 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    ----- Original Message -----
    From: "jcbollinger" <john.bollinger@stjude.org>
    To: puppet...@googlegroups.com <javascript:>
    Sent: Tuesday, October 22, 2013 3:13:14 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules


    On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:

    Another round of thanks for the replies to this thread. I apologize
    that
    almost as soon as I posted it, I got pulled off onto another project
    and
    wasn't able to follow up until now. Replies inline below, and there
    are
    probably a couple more coming to different branches (damn I miss
    Usenet
    threading!)

    John Bollinger wrote:
    We agree on most of what you said, but it doesn't seem very
    responsive
    to
    the comments to which they ostensibly reply. I am in no way
    arguing
    against the idea of the data in modules subsystem. It is a
    fantastic
    idea,
    and long past due. I *am* concerned, however, about the new
    approach
    Eric
    proposed. I suggested a more general approach than (my
    understanding
    of)
    the one he described, one not tied specifically to ::params
    classes.
    Inasmuch as you disfavor ::params classes, I would think that you
    would
    find much to like about my counterproposal. Indeed, I think my
    proposal is
    very much like the original prototype you floated.

    John I didn't see a more detailed description of what you're
    proposing;
    is
    this section (quoted from upthread) what you're referring to?
    Yes.

    Do I understand correctly that you set out to get rid of the
    ::params
    class pattern, but now you favor an approach that depends on that
    pattern?

    Heh, well when you put it that way...

    Let's also keep in mind that the purpose of the ::params class pattern
    is
    not really to serve as a per-module general data repository. Rather,
    it is
    specifically to provide a means for indirection of class parameter
    defaults. To the extent that ::params classes now do serve as data
    repositories, it is -- or should be -- in service to that purpose, not
    to a
    broader one. Data in modules is a *complementary*, but more general,
    approach whereby default values expressed in DSL code can in some
    cases be
    replaced by default values drawn from per-module data. Where data are
    consumed by a module in other ways or for other purposes, there is no
    particular reason why a ::params class should be involved.


    Why is that better than being more general: enable an implicit
    lowest-priority hierarchy level for values of form
    'modulename::variable',
    drawing on data from per-module data files such as
    modules/modulename/data.yaml?

    If I understand this correctly this is slightly different (and
    probably
    inadequate from RI's standpoint), because it just adds another
    'category'
    (in the ARM-9 sense) to the end of each lookup, and what RI and
    others
    propose is to have another _complete hiera invocation_ inside the
    module
    owning a class parameter's namespace the end of each unsuccessful
    site-hiera lookup. Separate hiera.yaml config file with its own
    hierarchy
    defined, and a tree of data files. (params.pp does this by letting
    old-school puppet DSL logic determine your "hierarchy")
    I don't have any particular objection to implementing data-in-modules
    as a
    separate full-fledged lookup against a per-module fallback hierarchy,
    but
    the qualitative differences from what I suggested are subtle. For the
    most
    part, I think it's just a question of how many levels you can or do
    add to
    the bottom of the logical hierarchy, whether it's implemented via one
    call
    to the hiera subsystem or two. There is a difference, however, in the
    behavior of lookups that collected data from across the hierarchy,
    i.e.
    hiera_hash() and hiera_array(). Those aren't relevant to class
    parameter
    binding (at this point), but it is worth considering what semantics
    are
    wanted there, and whether there might be a way for the caller to
    choose.

    I also talked to a user today who wants data from modules (by doing
    hash
    key merge on a parameter's class::subclass::varname) from *any*
    module in
    the modulepath to contribute, say, sudoers rules to the sudo module
    from
    other site-written modules that require particular sudoers stanzas.
    So
    I'm
    trying to consider how to pull all of this together without making a
    O(n^n)
    complexity explosion.

    I'm with R.I. in suggesting that you get something solid and
    fundamentally
    sound out soon, even if it doesn't address every user request on the
    first
    go (or ever). I understand how a confederated data source such as you
    now
    describe could be useful, but I think such a feature would require a
    significant effort in its own right.

    Furthermore, I think you are fast approaching the point where the data
    subsystem cannot automagically do the right thing in every case. I
    don't
    think it would be a sin to require some features to be explicitly
    declared
    or invoked by DSL code. For example, perhaps you want a data access
    function that allows the caller to somehow specify the scope of the
    data to
    search. Maybe a couple of releases down the line.
    Absolutely, there will never be a single data solution that solves 100% of
    problems
    but luckily we have multiple approaches - hiera, ENC, node terminii etc.
    This user
    can also write his own hiera backend to achieve his goal. Thats the point.
    Aim for a solid fit for the 80% of problems, inform and educate by the
    solution
    by being prescriptive but with extension points for the 20% of users who have
    really
    complex problems who can carry the cost of the complexity solving them
    brings.

    For the rest the design of the tool informs how they approach writing
    manifests
    and for green fields even how they build infrastructures.

    There is no chance that a 100% solution will be found for the data problem,
    just
    give up. Sometimes it's ok - as in this use case Eric mentions - to just say
    no
    that is not in the interest of the larger % of community and we're
    prioritizing
    shipping something over pleasing 100% of people.

    Just say no, point to extension points. ship something - as long as that
    something
    isn't impossible to extend for mortals like the arm9 work, because then you
    have to
    solve 100% of the problems.
    I have released a solution to this @
    http://www.devco.net/archives/2013/12/08/better-puppet-modules-using-hiera-data.php

    Need as wide as possible feedback and testing, lets move forward.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/8f174903-cbfc-4d11-831a-37658008f74a%40googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • R.I.Pienaar at Dec 9, 2013 at 12:37 am

    ----- Original Message -----
    From: "Alessandro Franceschi" <al@lab42.it>
    To: puppet-users@googlegroups.com
    Sent: Sunday, December 8, 2013 10:40:55 PM
    Subject: Re: [Puppet Users] Re: Status of Data in modules

    Wow, this looks promising: sane, plain and easy to use.
    Going to test it soon.
    Looking at things in perspective, how do you think this approach will go
    along with the one implemented in Puppet 3.3 ?
    In another thread it was mentioned by Hendrik that the 3.3.x one is being removed
    which prompted me to make my code into a module.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1052559863.511.1386549474054.JavaMail.zimbra%40devco.net.
    For more options, visit https://groups.google.com/groups/opt_out.
  • John Julien at Oct 16, 2013 at 2:37 am

    On Tuesday, October 15, 2013 8:08:10 AM UTC-5, jcbollinger wrote:


    I'm not saying that categorically getting rid of ::params classes is a
    viable target. In fact, I don't really understand why it was ever an
    objective in the first place. On the other hand, I don't see why it makes
    sense for Puppet to give special significance to that pattern, either. A
    more general data-in-modules feature such as I describe would give users
    the option to avoid ::params classes in some cases, and I'm inclined to
    think that it would be easier to implement, to understand, and to use.


    John
    I agree with this. It seems the proposed new solution adds complexity and
    non-intuitive value lookups that could confuse users. I think the
    data-in-modules feature sounds useful. Perhaps as John suggests a change
    in scope is needed. Instead of getting rid of params.pp the scope should
    be to move hierarchical data out of params.pp and leave variables derived
    from logic in params.pp.


    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • John Julien at Oct 16, 2013 at 2:45 am

    On Friday, October 11, 2013 7:12:32 PM UTC-5, Chuck wrote:
    What would be nice but not necessary:

    1) defined variable automatically have classname added to avoid global
    conflicts.


    eg.

    Module: apache

    variable: port

    becomes global hiera: apache::port
    +1
    Defining a variable in a module but having it's scope be global seems
    counter intuitive and could lead to conflicts. If someone wants a topscope
    variable they should probably define it in a topscope location (ENC,
    Centralized Hiera Files, etc)

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedOct 14, '13 at 2:16p
activeDec 9, '13 at 12:37a
posts20
users7
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase