FAQ
I have a difficult-to-manage application which does not implement a conf.d
or include syntax in its configuration, but requires a bunch of config
snippets that contain information only on groups of other servers. I've
been dealing with this by generating the config snippets from templates on
some servers as exported resources, realizing them on the central server,
and then executing an external script to "compile" these snippets into the
final config.

This has a couple of drawbacks. First, it requires puppet to stat nearly
15,000 little tiny config snippets every run that are not actually used
directly, and shouldn't need to exist. Second, the final config file,
because it's compiled by an external script, isn't under the control of
puppet, so it has no idea if that file gets modified by something outside..
so it can't know to update it.

I've been mulling over a better way to manage this config file, and I think
I've hit on an idea, but I have no idea if it will actually work, or what
the syntax would look like if it could.

I'm thinking of replacing the @@file resources on the remote servers with a
defined type .. say .. @@data_container. Then, on the server where the
data is needed I could use a collector to iterate over the exported
resources reading data from them to use in the single template for the
final config file.

Where the data is defined:
@@data_container { 'mydata':
    someparameter => 'foo'
}

And then in the template on the other host, somehow get a collection of
those resources into an array, and make use of their parameters as
variables to be referenced in the template..

<%- collection.each do |data| -%>
<%= data.someparameter %>
<%- end -%>

Would this work at all? Is there syntax to support something like this?


--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/7732833b-55f8-4e1b-8869-4ec231d36964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Daniel Siechniewicz at Sep 10, 2014 at 11:30 am
    Hi,

    Sounds like a job for https://github.com/dalen/puppet-puppetdbquery
    potentially? pdbresourcequery or maybe even the hiera backend.


    Regards,
    Daniel

    On Tuesday, September 9, 2014 11:49:21 PM UTC+1, mpou...@afilias.info wrote:

    I have a difficult-to-manage application which does not implement a conf.d
    or include syntax in its configuration, but requires a bunch of config
    snippets that contain information only on groups of other servers. I've
    been dealing with this by generating the config snippets from templates on
    some servers as exported resources, realizing them on the central server,
    and then executing an external script to "compile" these snippets into the
    final config.

    This has a couple of drawbacks. First, it requires puppet to stat nearly
    15,000 little tiny config snippets every run that are not actually used
    directly, and shouldn't need to exist. Second, the final config file,
    because it's compiled by an external script, isn't under the control of
    puppet, so it has no idea if that file gets modified by something outside..
    so it can't know to update it.

    I've been mulling over a better way to manage this config file, and I
    think I've hit on an idea, but I have no idea if it will actually work, or
    what the syntax would look like if it could.

    I'm thinking of replacing the @@file resources on the remote servers with
    a defined type .. say .. @@data_container. Then, on the server where the
    data is needed I could use a collector to iterate over the exported
    resources reading data from them to use in the single template for the
    final config file.

    Where the data is defined:
    @@data_container { 'mydata':
    someparameter => 'foo'
    }

    And then in the template on the other host, somehow get a collection of
    those resources into an array, and make use of their parameters as
    variables to be referenced in the template..

    <%- collection.each do |data| -%>
    <%= data.someparameter %>
    <%- end -%>

    Would this work at all? Is there syntax to support something like this?

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/59ba639d-441a-41f9-bcf4-93ac45fe209f%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matthew Pounsett at Sep 10, 2014 at 12:03 pm

    On 10 September 2014 07:30, Daniel Siechniewicz wrote:

    Hi,

    Sounds like a job for https://github.com/dalen/puppet-puppetdbquery
    potentially? pdbresourcequery or maybe even the hiera backend.
    Hiera doesn't apply here, because it's data gathered from the servers
    (mostly from facts) where the config snippets are defined. Putting that
    sort of thing in Hiera violates Don't Repeat Yourself. I'll have a look at
    the others though.. I'm not familiar with them. Thanks for the suggestions.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAKZ22fLmwD0eqXs7DttKydpZjPBTJMJq0PcQY28vpnbO%3D7mxGg%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Daniel Siechniewicz at Sep 10, 2014 at 12:08 pm
    Hi,

    This particular hiera backend, from what I understand, is extracting
    values from puppetdb "on the fly", so you don't have to put any values
    in hiera yaml/json files save for a relevant puppetdb query. This
    would centralize data gathering and localize it to a "data component".
    Otherwise you end up with puppetdb queries within your manifests which
    could be not what you want. Then again, this really depends on your
    preferences. Good luck and tell us if this worked for you.


    Regards,
    Daniel

    On 10 September 2014 13:03, Matthew Pounsett wrote:

    On 10 September 2014 07:30, Daniel Siechniewicz wrote:

    Hi,

    Sounds like a job for https://github.com/dalen/puppet-puppetdbquery
    potentially? pdbresourcequery or maybe even the hiera backend.

    Hiera doesn't apply here, because it's data gathered from the servers
    (mostly from facts) where the config snippets are defined. Putting that
    sort of thing in Hiera violates Don't Repeat Yourself. I'll have a look at
    the others though.. I'm not familiar with them. Thanks for the suggestions.

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/puppet-users/VzW7ZODB7hQ/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/CAKZ22fLmwD0eqXs7DttKydpZjPBTJMJq0PcQY28vpnbO%3D7mxGg%40mail.gmail.com.

    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2BSnNS-XdeWFXeJy75LAnn5Smb9u1R0vvBSJz_ZUB3Ta-paOkQ%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matthew Pounsett at Sep 10, 2014 at 2:09 pm

    On 10 September 2014 08:08, Daniel Siechniewicz wrote:

    Hi,

    This particular hiera backend, from what I understand, is extracting
    values from puppetdb "on the fly", so you don't have to put any values
    in hiera yaml/json files save for a relevant puppetdb query. This
    would centralize data gathering and localize it to a "data component".
    Otherwise you end up with puppetdb queries within your manifests which
    could be not what you want. Then again, this really depends on your
    preferences. Good luck and tell us if this worked for you.

    It looks like it's designed to deal with facts and classes only. I really
    need something that can get data from resources (defined types). It's
    crappy design, but I could move *some* of the global variables defined in
    my type's module into facts to make those available, but I don't see a
    decent way to get data out of the resources, so I don't think it solves the
    whole problem.

    Thanks for the suggestion though.. this looks like it could be very useful
    to simplify a couple other things I've been looking at.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAKZ22fLvPWyy4rcaiDEWq1tda777hTAf6PBAXBJKgmjS627s9Q%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Jcbollinger at Sep 15, 2014 at 2:26 pm

    On Wednesday, September 10, 2014 9:09:14 AM UTC-5, Matthew Pounsett wrote:


    On 10 September 2014 08:08, Daniel Siechniewicz <dan...@siechniewicz.com
    <javascript:>> wrote:
    Hi,

    This particular hiera backend, from what I understand, is extracting
    values from puppetdb "on the fly", so you don't have to put any values
    in hiera yaml/json files save for a relevant puppetdb query. This
    would centralize data gathering and localize it to a "data component".
    Otherwise you end up with puppetdb queries within your manifests which
    could be not what you want. Then again, this really depends on your
    preferences. Good luck and tell us if this worked for you.

    It looks like it's designed to deal with facts and classes only. I
    really need something that can get data from resources (defined types).

    Or maybe that's just how you've determined to do it. And it is indeed a
    crappy design, because Puppet resources -- including exported ones -- are
    not intended to function as data vehicles within the master, and they do
    not conveniently do so. Instead, they are intended to carry data from
    master to agent.


    It's crappy design, but I could move *some* of the global variables
    defined in my type's module into facts to make those available, but I don't
    see a decent way to get data out of the resources, so I don't think it
    solves the whole problem.
    If you have to manage a site configuration wherein one machine's
    configuration needs to draw on details of many other machines', then it
    seems to me that the way to go would be to model all that data in hiera in
    such a way that each machine can get what it needs from there. The main
    thing there is to avoid putting anything in a machine-specific hierarchy
    level if more than one node needs to draw on it. You can use hashes keyed
    on the node identifier instead.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f11d7e12-45b2-4871-8bf6-b31184c15554%40googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • José Luis Ledesma at Sep 10, 2014 at 4:57 pm
    Perhaps you can use datacat_fragments exported resources and then collect
    them and put them together with the datacat_collector.

    https://github.com/richardc/puppet-datacat/

    Regards,
    El 10/09/2014 00:49, <mpounsett@afilias.info> escribió:
    I have a difficult-to-manage application which does not implement a conf.d
    or include syntax in its configuration, but requires a bunch of config
    snippets that contain information only on groups of other servers. I've
    been dealing with this by generating the config snippets from templates on
    some servers as exported resources, realizing them on the central server,
    and then executing an external script to "compile" these snippets into the
    final config.

    This has a couple of drawbacks. First, it requires puppet to stat nearly
    15,000 little tiny config snippets every run that are not actually used
    directly, and shouldn't need to exist. Second, the final config file,
    because it's compiled by an external script, isn't under the control of
    puppet, so it has no idea if that file gets modified by something outside..
    so it can't know to update it.

    I've been mulling over a better way to manage this config file, and I
    think I've hit on an idea, but I have no idea if it will actually work, or
    what the syntax would look like if it could.

    I'm thinking of replacing the @@file resources on the remote servers with
    a defined type .. say .. @@data_container. Then, on the server where the
    data is needed I could use a collector to iterate over the exported
    resources reading data from them to use in the single template for the
    final config file.

    Where the data is defined:
    @@data_container { 'mydata':
    someparameter => 'foo'
    }

    And then in the template on the other host, somehow get a collection of
    those resources into an array, and make use of their parameters as
    variables to be referenced in the template..

    <%- collection.each do |data| -%>
    <%= data.someparameter %>
    <%- end -%>

    Would this work at all? Is there syntax to support something like this?


    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit
    https://groups.google.com/d/msgid/puppet-users/7732833b-55f8-4e1b-8869-4ec231d36964%40googlegroups.com
    <https://groups.google.com/d/msgid/puppet-users/7732833b-55f8-4e1b-8869-4ec231d36964%40googlegroups.com?utm_medium=email&utm_source=footer>
    .
    For more options, visit https://groups.google.com/d/optout.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAF_B3dckc5ddaL%2BMbGokiQ6r1za1RZx8E7vw5VmszPOhMQ1jLg%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.
  • Matthew Pounsett at Sep 10, 2014 at 5:05 pm

    On 10 September 2014 12:57, José Luis Ledesma wrote:

    Perhaps you can use datacat_fragments exported resources and then collect
    them and put them together with the datacat_collector.

    https://github.com/richardc/puppet-datacat/
    That looks really promising. It'll take me a bit to wrap my head around
    it, because I think I'll need to refactor a couple of things to use it...
    but I think I can make it do what I need.

    Thanks!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAKZ22f%2BnuKETYmKLxeRh7wdU%3DTuzizM5preHJThCgN8%3DABQPEA%40mail.gmail.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedSep 9, '14 at 10:49p
activeSep 15, '14 at 2:26p
posts8
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase