FAQ
Hi,

I've written a set of Puppet modules to manage servers, but am
slightly concerned about the structure I've used as it seems to differ
significantly from anything else I've seen in Puppet Forge/elsewhere
on the internet.

I've made extensive use of definitions so that defaults for modules
can be overridden in the node manifest for a server. For example:

node examplenode.example.com {

include apache
include mysql

define apache::config {
listenport => 8080
}

define mysql::config
}

So in each module, I have a config definition which must be called in
each node manifest, even if you don't want to override any defaults
(as in the case of MySQL above). I haven't used parametrized classes
at all.

It all works fine, but are there any disadvantages to the sort of
structure I'm using? Should I be using more parametrized classes
instead of definitions?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Jcbollinger at Apr 24, 2012 at 4:13 pm

    On Apr 24, 9:13 am, Andy Taylor wrote:
    Hi,

    I've written a set of Puppet modules to manage servers, but am
    slightly concerned about the structure I've used as it seems to differ
    significantly from anything else I've seen in Puppet Forge/elsewhere
    on the internet.

    I've made extensive use of definitions so that defaults for modules
    can be overridden in the node manifest for a server. For example:

    node examplenode.example.com {

    include apache
    include mysql

    define apache::config {
    listenport => 8080

    }

    define mysql::config

    }

    So in each module, I have a config definition which must be called in
    each node manifest, even if you don't want to override any defaults
    (as in the case of MySQL above). I haven't used parametrized classes
    at all.

    It all works fine,

    That it works well is the most important criterion. With that
    said, ...

    but are there any disadvantages to the sort of
    structure I'm using? Should I be using more parametrized classes
    instead of definitions?

    One problem with using definitions, especially definitions that are
    required to be instantiated for multiple classes to work, is that each
    instantiation of a definition is a separate resource. Each must
    therefore have a unique name among the resources of its type, and each
    will be applied separately. This presents either a management problem
    (who is responsible for instantiating the definition in any given set
    of circumstances?) or an efficiency problem (equivalent definition
    instances are redundantly applied) or both.

    Parameterized classes do not present that efficiency problem because
    all classes are singletons. That intensifies the management problem,
    however, because unlike ordinary classes, each parameterized class can
    be declared only once for any given node. In general, few problems
    are best solved by parameterized classes.

    You should use defined types only when you need *resources* for which
    there is no native type available. In that case you should know that
    you want resources rather than classes. That might be because nodes
    can have more than one instance of your 'thing', because you want
    support for virtual or exported 'things', or (best) simply because
    your 'thing' is better characterized as something nodes *have* than as
    something that they *are*.

    On the other hand, you should use (ordinary) classes where you want
    their singleton nature -- especially idempotency, which is a far more
    useful property than many appreciate. You must also use classes where
    you need your 'thing' to be declared via an ENC (the C is for
    CLASSifier, after all). You should generally use classes where they
    represent an aspect of nodes' type / kind / role / nature, as opposed
    to a tangible thing you can actually manage.

    As for default values and value overriding, if your 'things' are a
    better fit for classes than for definitions, then you should be
    looking at external data and Hiera instead of parameterization. Even
    for any definitions you retain, it might be to your advantage to
    define certain default values via external data than to code them into
    your manifests, or even to remove parameters in favor of external
    data. Really, it is best if your manifests contain no node- or site-
    specific data.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Andy Taylor at Apr 24, 2012 at 5:31 pm
    Thank you very much for your detailed reply John. I hadn't looked at
    Hiera before, it looks very interesting. On the point of it's not good
    to include node/site data in manifests, it's kind of essential in my
    eyes to the setup I am trying to implement.

    At my company, one server may serve a site which has very different
    requirements to another, so I really needed something that was very
    flexible. To expand on the structure I outlined in my previous post:
    each server has it's own node.pp file. If you need to change something
    (e.g. MySQL settings, which will probably the most frequent one that
    needs changing), you can simply edit that server's node manifest file.
    The great advantage here is that each site's node manifest serves as
    documentation of what changes have been made to it. It's also very
    easy to read and understand for someone new to Puppet.

    From some quick reading of the Hiera posts on the Puppet Labs blog, I
    guess I could simply have a yaml file for each server with overrides
    instead of doing it via the node manifest. However, I don't really see
    the advantage of this approach, except for it would cut out some of
    the clutter in my Puppet modules. The idea of a hierarchical set of
    data being considered in line with Facter facts is very interesting
    though, especially if I combined it with some custom facts of my own;
    I will do some more reading on that.

    With the class/definition point, I am wherever possible using classes,
    as usually I'm dealing with the nature of a node (e.g. you are a
    webserver, database server etc.) The only case I have used definitions
    is for these configuration file overrides. So if I'm understanding
    your comments on class/definition use properly, I think I'm sort of
    taking the right approach to it...

    The key concern for me at the moment is flexibility and scalability...
    none of my stuff is live yet, and I want to make sure I'm using best
    practices before I put it all in place. :)
    On Apr 24, 5:13 pm, jcbollinger wrote:
    On Apr 24, 9:13 am, Andy Taylor wrote:








    Hi,
    I've written a set of Puppet modules to manage servers, but am
    slightly concerned about the structure I've used as it seems to differ
    significantly from anything else I've seen in Puppet Forge/elsewhere
    on the internet.
    I've made extensive use of definitions so that defaults for modules
    can be overridden in the node manifest for a server. For example:
    node examplenode.example.com {
    include apache
    include mysql
    define apache::config {
    listenport => 8080
    }
    define mysql::config
    }
    So in each module, I have a config definition which must be called in
    each node manifest, even if you don't want to override any defaults
    (as in the case of MySQL above). I haven't used parametrized classes
    at all.
    It all works fine,
    That it works well is the most important criterion.  With that
    said, ...
    but are there any disadvantages to the sort of
    structure I'm using? Should I be using more parametrized classes
    instead of definitions?
    One problem with using definitions, especially definitions that are
    required to be instantiated for multiple classes to work, is that each
    instantiation of a definition is a separate resource.  Each must
    therefore have a unique name among the resources of its type, and each
    will be applied separately.  This presents either a management problem
    (who is responsible for instantiating the definition in any given set
    of circumstances?) or an efficiency problem (equivalent definition
    instances are redundantly applied) or both.

    Parameterized classes do not present that efficiency problem because
    all classes are singletons.  That intensifies the management problem,
    however, because unlike ordinary classes, each parameterized class can
    be declared only once for any given node.  In general, few problems
    are best solved by parameterized classes.

    You should use defined types only when you need *resources* for which
    there is no native type available.  In that case you should know that
    you want resources rather than classes.  That might be because nodes
    can have more than one instance of your 'thing', because you want
    support for virtual or exported 'things', or (best) simply because
    your 'thing' is better characterized as something nodes *have* than as
    something that they *are*.

    On the other hand, you should use (ordinary) classes where you want
    their singleton nature -- especially idempotency, which is a far more
    useful property than many appreciate.  You must also use classes where
    you need your 'thing' to be declared via an ENC (the C is for
    CLASSifier, after all).  You should generally use classes where they
    represent an aspect of nodes' type / kind / role / nature, as opposed
    to a tangible thing you can actually manage.

    As for default values and value overriding, if your 'things' are a
    better fit for classes than for definitions, then you should be
    looking at external data and Hiera instead of parameterization.  Even
    for any definitions you retain, it might be to your advantage to
    define certain default values via external data than to code them into
    your manifests, or even to remove parameters in favor of external
    data.  Really, it is best if your manifests contain no node- or site-
    specific data.

    John
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jcbollinger at Apr 25, 2012 at 1:53 pm

    On Apr 24, 12:31 pm, Andy Taylor wrote:
    Thank you very much for your detailed reply John. I hadn't looked at
    Hiera before, it looks very interesting. On the point of it's not good
    to include node/site data in manifests, it's kind of essential in my
    eyes to the setup I am trying to implement.

    Unless the setup you're trying to implement is "one with node-specific
    data in its manifests", you are mistaken. Or perhaps you
    misunderstand me. I am not at all saying that your manifests should
    not rely on node- or site-specific data, only that those data do not
    belong in the manifest files themselves. Instead, put them in an
    external data source, such as one accessed via Hiera. That's why I
    mentioned the principle in the first place.

    At my company, one server may serve a site which has very different
    requirements to another, so I really needed something that was very
    flexible. To expand on the structure I outlined in my previous post:
    each server has it's own node.pp file. If you need to change something
    (e.g. MySQL settings, which will probably the most frequent one that
    needs changing), you can simply edit that server's node manifest file.
    The great advantage here is that each site's node manifest serves as
    documentation of what changes have been made to it. It's also very
    easy to read and understand for someone new to Puppet.

    It is somewhat better for all node-specific data to be associated with
    nodes' declarations, and for all site-specific data to be in site.pp,
    than for such data to be spread throughout your manifests. It still
    presents problems, however.

    In particular, it's brittle to put the node-specific data into global
    variable declarations. The only way that can work is to use
    conditional 'import'ing of the appropriate node file (else different
    nodes' declarations will clash). 'Import' itself is best avoided as
    much as possible, though you'd be on the borderline of one of the few
    reasonable use cases.

    On the other hand, putting node-specific data into node-scoped
    variables limits you to either relying on dynamic scoping (which is
    being phased out) or using those variables only directly inside your
    node declaration.

    Using literal node-specific values directly as function, class, or
    defined type parameters is a bit better, but it requires you to
    forward those values throughout the rest of your manifests wherever
    they are needed. That complicates everything, requiring extra
    parameters, and perhaps even preventing you from using non-
    parameterized classes in some places.

    From some quick reading of the Hiera posts on the Puppet Labs blog, I
    guess I could simply have a yaml file for each server with overrides
    instead of doing it via the node manifest.

    Pretty much, yes.

    However, I don't really see
    the advantage of this approach, except for it would cut out some of
    the clutter in my Puppet modules.

    More than that:

    1) It could greatly simplify and clarify some of your manifests. Your
    node declarations especially would benefit, but reducing
    parameterization elsewhere clarifies your manifests by focusing
    attention on the big picture -- roles, features, capabilities -- and
    away from the details.

    2) It can help you obtain the benefits of using classes (where they
    are appropriate) without requiring class parameterization (which would
    lose you some of those same benefits).

    3) From a change management perspective, separating the data allows
    you to more easily recognize the nature of any given change.

    The idea of a hierarchical set of
    data being considered in line with Facter facts is very interesting
    though, especially if I combined it with some custom facts of my own;
    I will do some more reading on that.

    With the class/definition point, I am wherever possible using classes,
    as usually I'm dealing with the nature of a node (e.g. you are a
    webserver, database server etc.) The only case I have used definitions
    is for these configuration file overrides. So if I'm understanding
    your comments on class/definition use properly, I think I'm sort of
    taking the right approach to it...

    It sounds like you are generally doing the right thing, but you did
    ask about those definitions, and I don't think they're good style. As
    I said before, however, the most important criterion is that it works
    well.

    The key concern for me at the moment is flexibility and scalability...
    none of my stuff is live yet, and I want to make sure I'm using best
    practices before I put it all in place. :)

    I think making your manifests as generic as possible, separating data
    from manifests, and using a flexible external data source such as
    Hiera is a win on both flexibility and scalability. I doubt you'll
    find a consensus on best practices, but I am confident that you will
    find others who also consider those to be Puppet best-practices
    principles.


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ramin K at Apr 25, 2012 at 6:01 pm

    On 4/24/2012 10:31 AM, Andy Taylor wrote:
    Thank you very much for your detailed reply John. I hadn't looked at
    Hiera before, it looks very interesting. On the point of it's not good
    to include node/site data in manifests, it's kind of essential in my
    eyes to the setup I am trying to implement.

    At my company, one server may serve a site which has very different
    requirements to another, so I really needed something that was very
    flexible. To expand on the structure I outlined in my previous post:
    each server has it's own node.pp file. If you need to change something
    (e.g. MySQL settings, which will probably the most frequent one that
    needs changing), you can simply edit that server's node manifest file.
    The great advantage here is that each site's node manifest serves as
    documentation of what changes have been made to it. It's also very
    easy to read and understand for someone new to Puppet.

    From some quick reading of the Hiera posts on the Puppet Labs blog, I
    guess I could simply have a yaml file for each server with overrides
    instead of doing it via the node manifest. However, I don't really see
    the advantage of this approach, except for it would cut out some of
    the clutter in my Puppet modules. The idea of a hierarchical set of
    data being considered in line with Facter facts is very interesting
    though, especially if I combined it with some custom facts of my own;
    I will do some more reading on that.

    With the class/definition point, I am wherever possible using classes,
    as usually I'm dealing with the nature of a node (e.g. you are a
    webserver, database server etc.) The only case I have used definitions
    is for these configuration file overrides. So if I'm understanding
    your comments on class/definition use properly, I think I'm sort of
    taking the right approach to it...

    The key concern for me at the moment is flexibility and scalability...
    none of my stuff is live yet, and I want to make sure I'm using best
    practices before I put it all in place. :)
    Mr Bollinger did his usual excellent job of thoroughly explaining the
    problem space in other parts of this thread. I'd simply add that using a
    richer data store, such as Hiera, can keep you from doing complicated
    gymnastics in your module code.

    I'm using a hiera.yaml file similar to the following with fqdn at the
    top and common at the bottom. I've added a role fact which is reading
    /etc/role on the client. You can use any Facter fact such as %{domain}
    or write your own. %{location} or %{datacenter} seems to be a common one
    for admins with multiple sites.

    :hierarchy:
    - %{fqdn}
    - %{role}_%{environment}
    - %{role}
    - %{environment}
    - common

    In my mysql::data class I add a variable and give it a default of 256M.

    $innodb_buffer_pool_size = hiera('mysql_innodb_buffer_pool_size', '256M')

    I can then set the following to deal with the specifics of my environment.
    512M in my prod environment, production.yaml
    4G in role db, db.yaml
    1.5G on role db in environment stage, db_stage.yaml
    8G on server db01.main.sfo, db01.main.sfo.mydomain.com.yaml

    Going one step further you can use hiera_array to merge data. For
    example you might allow a set of servers to monitor all servers and then
    allow the local monitors as well. Assuming a location fact and the
    following data files

    common.yaml
    monitoring_hosts:
    - 'mon01.ord'
    - 'mon02.ord'

    lax.yaml
    monitoring_hosts:
    - 'mon01.lax'

    class someclass::data {
    $hosts = hiera_array('monitoring_hosts')
    }

    someclass::data::hosts = ['mon01.lax','mon01.ord','mon02.ord']

    Hopefully this should give you some ideas of how to use Hiera in your
    new system.

    Ramin

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Andy Taylor at Apr 27, 2012 at 3:04 pm
    Thanks John for yet another very useful reply, it is much appreciated.
    I'm going to do some more reading on Hiera and look at reworking my
    modules.

    And thank you Ramin for some examples of Hiera in practice, looks very
    interesting! Going to have a play around with it next week :)

    Thanks!
    On Apr 25, 7:01 pm, Ramin K wrote:
    On 4/24/2012 10:31 AM,AndyTaylorwrote:








    Thank you very much for your detailed reply John. I hadn't looked at
    Hiera before, it looks very interesting. On the point of it's not good
    to include node/site data in manifests, it's kind of essential in my
    eyes to the setup I am trying to implement.
    At my company, one server may serve a site which has very different
    requirements to another, so I really needed something that was very
    flexible. To expand on the structure I outlined in my previous post:
    each server has it's own node.pp file. If you need to change something
    (e.g. MySQL settings, which will probably the most frequent one that
    needs changing), you can simply edit that server's node manifest file.
    The great advantage here is that each site's node manifest serves as
    documentation of what changes have been made to it. It's also very
    easy to read and understand for someone new to Puppet.
    From some quick reading of the Hiera posts on the Puppet Labs blog, I
    guess I could simply have a yaml file for each server with overrides
    instead of doing it via the node manifest. However, I don't really see
    the advantage of this approach, except for it would cut out some of
    the clutter in my Puppet modules. The idea of a hierarchical set of
    data being considered in line with Facter facts is very interesting
    though, especially if I combined it with some custom facts of my own;
    I will do some more reading on that.
    With the class/definition point, I am wherever possible using classes,
    as usually I'm dealing with the nature of a node (e.g. you are a
    webserver, database server etc.) The only case I have used definitions
    is for these configuration file overrides. So if I'm understanding
    your comments on class/definition use properly, I think I'm sort of
    taking the right approach to it...
    The key concern for me at the moment is flexibility and scalability...
    none of my stuff is live yet, and I want to make sure I'm using best
    practices before I put it all in place. :)
    Mr Bollinger did his usual excellent job of thoroughly explaining the
    problem space in other parts of this thread. I'd simply add that using a
    richer data store, such as Hiera, can keep you from doing complicated
    gymnastics in your module code.

    I'm using a hiera.yaml file similar to the following with fqdn at the
    top and common at the bottom. I've added a role fact which is reading
    /etc/role on the client. You can use any Facter fact such as %{domain}
    or write your own. %{location} or %{datacenter} seems to be a common one
    for admins with multiple sites.

    :hierarchy:
    - %{fqdn}
    - %{role}_%{environment}
    - %{role}
    - %{environment}
    - common

    In my mysql::data class I add a variable and give it a default of 256M.

    $innodb_buffer_pool_size = hiera('mysql_innodb_buffer_pool_size', '256M')

    I can then set the following to deal with the specifics of my environment.
    512M in my prod environment, production.yaml
    4G in role db, db.yaml
    1.5G on role db in environment stage, db_stage.yaml
    8G on server db01.main.sfo, db01.main.sfo.mydomain.com.yaml

    Going one step further you can use hiera_array to merge data. For
    example you might allow a set of servers to monitor all servers and then
    allow the local monitors as well. Assuming a location fact and the
    following data files

    common.yaml
    monitoring_hosts:
    - 'mon01.ord'
    - 'mon02.ord'

    lax.yaml
    monitoring_hosts:
    - 'mon01.lax'

    class someclass::data {
    $hosts = hiera_array('monitoring_hosts')

    }

    someclass::data::hosts = ['mon01.lax','mon01.ord','mon02.ord']

    Hopefully this should give you some ideas of how to use Hiera in your
    new system.

    Ramin
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedApr 24, '12 at 2:13p
activeApr 27, '12 at 3:04p
posts6
users3
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase