FAQ
Hello folks,

After some conversation on #puppet on Freenode IRC, Eric Sorenson
requested I repost the information and question here, so I am doing so
and hopefully it will all make sense...

We currently have a well-established and relatively complex Puppet
setup in place at my company and I'm in the process of trying to
streamline changes as well as implement better testing to ensure
minimal disruption or issues when making those changes. Some
information on the current situation:

- There are currently three environments: development, staging,
production. These are controlled via the '--environment' setting for
puppet in each client. All clients only belong to one environment and
do not move between them.
- We have a single Puppet configuration to manage all environments.
Various conditional statements based on environment, application type,
hostname, etc. control what each client receives for its
configuration.
- There are separate servers for each environment for security reasons
(primarily sensitive information that can only exist in the production
environment).
- The Puppet configuration maintained via a Git repo, currently on a
single branch.
- Each person on the admin team checks out own copy of the repo, make
changes, commits the changes, then updates each environment on the
Puppet servers for the changes to take effect.

There are several issues with this process, unfortunately:

- Every so often a configuration mistake will adversely affect an
entire environment, and much of the time is only noticed _after_ the
changes are pushed out. As a result, local changes tend to be made in
the development environment for testing and sometimes aren't committed
for a long time, leaving discrepancies between the environments which
can lead to other subtle issues.
- Less frequent but still occuring often enough, changes can still
have subtle issues which cause things to work in one environment and
break horribly in another; this is especially bad when the broken
environment is the production one.
- The configuration for a given type of client is complex enough that
to change a client to a different application type (what we primarily
key most of our configuration off of, followed by the environment) to
test against a server would require rebuilding the client, which is a
25-45 minute process; too slow for simple changes and even too slow
for all but the most complex changes, given how many changes we make
in a single day.
- We allow our users to create local VMs that the development Puppet
server can key off their names to create a given configuration, but
since the configuration for the various environments is shared in a
Puppet configuration, potential for users point their puppet agents to
the production environment is a concern (due to the sensitive
information there).

After discussion with a few coworkers, the following process was laid
out to try to implement to resolve these issues:

- Create separate branches for each of the environments and have only
the matching branch checked out on the primary Puppet servers; changes
will be merged into the various branches one at a time to prevent
unintentional changes in a given environment before testing can be
done on that environment.
- Ensure a client in a given environment can ONLY run against that
configuration (e.g. disallow a client in the development environment
requesting the production configuration).
- Each person on the admin team will have a test server where they can
create their own branches from the Git repo for the changes they're
working and use their test server to test changes against existing
clients in the various environments (preventing the need to build out
a new test client(s) to validate each change). The existing clients
would only be run in no-op mode against the test servers.

The reason for each person on the admin team to have their own test
server that has access to all the environments is considered since:

- Having a different server for each environment would be affected by
the tight hardware resources currently.
- The need for having separate test servers would prevent needing to
use the primary servers for testing, which is difficult due to
multiple admins continuously making changes and needing to test them
without disturbing the other admins' work, along with not affecting
the current primary servers from being able to properly handle their
existing clients.

What this all boils down to is I'm trying to find a way to deal with a
single test server trying to be able to communicate with existing
clients in all the environments; most of the current configuration
would work fine except for the cert issue, which is the sticking point
at this time. Any solution on how to handle this in the most
straightforward manner would greatly be appreciated, as my research
has been leading to solutions far more complex than what I would like
(such as load balancing for the CA or trying to synchronize the certs
across the various systems).

Hopefully this made sense and I can find someone who can give advice
on how to proceed with this issue. Thanks in advance.

- Ken Lareau

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Eric Sorenson at Apr 18, 2012 at 2:57 pm
    Hi Ken, thanks for posting.

    It seems like you have introduced some tension between the security
    requirements (clients which are in a particular environment must not be
    able to retrieve other environments) and the need to have widespread
    testing with good coverage. From what I understand you've managed this now
    by having different CA certificates for each environment, but -- as it
    sounds like you realise -- that is pretty problematic. I'd suggest you can
    end-run around a lot of this trouble by using an external node classifier
    to set and enforce client environment, so regardless of what the
    --environment string says from the client, each of the masters will
    consistently enforce policy. You don't have to go all-in to node
    class/parameter assignment in the ENC because static configs are merged
    with the output. A prerequisite is that you need to have some source of
    truth which the ENCs can consult to determine the disposition of each node
    that connects, but that source-of-truth could be an access-controlled
    database (maybe you already have one?) that is, in general, going to be a
    better place to put business logic (which nodes should be able to access
    each environment, and even perhaps some of the conditional logic in your
    manifests) than Puppet itself.

    This isn't without its own set of problems and might not be a panacea but I
    think would help a lot of your use case. Some relevant reading:
    http://docs.puppetlabs.com/guides/external_nodes.html
    https://projects.puppetlabs.com/issues/3910
    https://projects.puppetlabs.com/issues/12869

    Hope this helps
    - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook -
    On Tuesday, April 17, 2012 7:34:43 PM UTC-7, Ken Lareau wrote:

    Hello folks,

    After some conversation on #puppet on Freenode IRC, Eric Sorenson
    requested I repost the information and question here, so I am doing so
    and hopefully it will all make sense...

    We currently have a well-established and relatively complex Puppet
    setup in place at my company and I'm in the process of trying to
    streamline changes as well as implement better testing to ensure
    minimal disruption or issues when making those changes. Some
    information on the current situation:

    - There are currently three environments: development, staging,
    production. These are controlled via the '--environment' setting for
    puppet in each client. All clients only belong to one environment and
    do not move between them.
    - We have a single Puppet configuration to manage all environments.
    Various conditional statements based on environment, application type,
    hostname, etc. control what each client receives for its
    configuration.
    - There are separate servers for each environment for security reasons
    (primarily sensitive information that can only exist in the production
    environment).
    - The Puppet configuration maintained via a Git repo, currently on a
    single branch.
    - Each person on the admin team checks out own copy of the repo, make
    changes, commits the changes, then updates each environment on the
    Puppet servers for the changes to take effect.

    There are several issues with this process, unfortunately:

    - Every so often a configuration mistake will adversely affect an
    entire environment, and much of the time is only noticed _after_ the
    changes are pushed out. As a result, local changes tend to be made in
    the development environment for testing and sometimes aren't committed
    for a long time, leaving discrepancies between the environments which
    can lead to other subtle issues.
    - Less frequent but still occuring often enough, changes can still
    have subtle issues which cause things to work in one environment and
    break horribly in another; this is especially bad when the broken
    environment is the production one.
    - The configuration for a given type of client is complex enough that
    to change a client to a different application type (what we primarily
    key most of our configuration off of, followed by the environment) to
    test against a server would require rebuilding the client, which is a
    25-45 minute process; too slow for simple changes and even too slow
    for all but the most complex changes, given how many changes we make
    in a single day.
    - We allow our users to create local VMs that the development Puppet
    server can key off their names to create a given configuration, but
    since the configuration for the various environments is shared in a
    Puppet configuration, potential for users point their puppet agents to
    the production environment is a concern (due to the sensitive
    information there).

    After discussion with a few coworkers, the following process was laid
    out to try to implement to resolve these issues:

    - Create separate branches for each of the environments and have only
    the matching branch checked out on the primary Puppet servers; changes
    will be merged into the various branches one at a time to prevent
    unintentional changes in a given environment before testing can be
    done on that environment.
    - Ensure a client in a given environment can ONLY run against that
    configuration (e.g. disallow a client in the development environment
    requesting the production configuration).
    - Each person on the admin team will have a test server where they can
    create their own branches from the Git repo for the changes they're
    working and use their test server to test changes against existing
    clients in the various environments (preventing the need to build out
    a new test client(s) to validate each change). The existing clients
    would only be run in no-op mode against the test servers.

    The reason for each person on the admin team to have their own test
    server that has access to all the environments is considered since:

    - Having a different server for each environment would be affected by
    the tight hardware resources currently.
    - The need for having separate test servers would prevent needing to
    use the primary servers for testing, which is difficult due to
    multiple admins continuously making changes and needing to test them
    without disturbing the other admins' work, along with not affecting
    the current primary servers from being able to properly handle their
    existing clients.

    What this all boils down to is I'm trying to find a way to deal with a
    single test server trying to be able to communicate with existing
    clients in all the environments; most of the current configuration
    would work fine except for the cert issue, which is the sticking point
    at this time. Any solution on how to handle this in the most
    straightforward manner would greatly be appreciated, as my research
    has been leading to solutions far more complex than what I would like
    (such as load balancing for the CA or trying to synchronize the certs
    across the various systems).

    Hopefully this made sense and I can find someone who can give advice
    on how to proceed with this issue. Thanks in advance.

    - Ken Lareau
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/HvWvZVO4JbYJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ken Lareau at Apr 18, 2012 at 8:14 pm
    Eric,

    Thank you for the response, and yes, our current configuration and security
    requirements have made things a bit difficult at the moment. Fortunately
    we do already have an ENC which does access an access-controlled database
    and does have the environment information in it, though we still do pass
    '--environment' due to this not working long ago... and looking at the
    first issue ticket you mention, this could be a problem as our developers
    are allowed to connect their own VMs to the development Puppet server but
    can easily choose to point '--environment' to whatever they please. In
    actuality, they can do that now (and it would probably work), so the
    problem is already there, though I would definitely not want to make it
    worse. :) Right now our ENC only has a minimal amount of information in
    it, but the plan is to eventually populate it with more and reduce the
    amount of work the Puppet configuration has to do itself, as you suggested
    below.

    From the brief exchange on IRC this morning, the indication seems to be
    moving to a single CA should be the follow-up to this, and I think this is
    doable, though I'm still uncertain what the best path is to handle this.
    Once that's done, I can then look into improving the security of our
    systems (as in actually making it secure rather than what it really is
    right now).

    Thank you for your input.

    - Ken Lareau

    On Wed, Apr 18, 2012 at 7:56 AM, Eric Sorenson wrote:

    Hi Ken, thanks for posting.

    It seems like you have introduced some tension between the security
    requirements (clients which are in a particular environment must not be
    able to retrieve other environments) and the need to have widespread
    testing with good coverage. From what I understand you've managed this now
    by having different CA certificates for each environment, but -- as it
    sounds like you realise -- that is pretty problematic. I'd suggest you can
    end-run around a lot of this trouble by using an external node classifier
    to set and enforce client environment, so regardless of what the
    --environment string says from the client, each of the masters will
    consistently enforce policy. You don't have to go all-in to node
    class/parameter assignment in the ENC because static configs are merged
    with the output. A prerequisite is that you need to have some source of
    truth which the ENCs can consult to determine the disposition of each node
    that connects, but that source-of-truth could be an access-controlled
    database (maybe you already have one?) that is, in general, going to be a
    better place to put business logic (which nodes should be able to access
    each environment, and even perhaps some of the conditional logic in your
    manifests) than Puppet itself.

    This isn't without its own set of problems and might not be a panacea but
    I think would help a lot of your use case. Some relevant reading:
    http://docs.puppetlabs.com/guides/external_nodes.html
    https://projects.puppetlabs.com/issues/3910
    https://projects.puppetlabs.com/issues/12869

    Hope this helps
    - Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook -
    On Tuesday, April 17, 2012 7:34:43 PM UTC-7, Ken Lareau wrote:

    Hello folks,

    After some conversation on #puppet on Freenode IRC, Eric Sorenson
    requested I repost the information and question here, so I am doing so
    and hopefully it will all make sense...

    We currently have a well-established and relatively complex Puppet
    setup in place at my company and I'm in the process of trying to
    streamline changes as well as implement better testing to ensure
    minimal disruption or issues when making those changes. Some
    information on the current situation:

    - There are currently three environments: development, staging,
    production. These are controlled via the '--environment' setting for
    puppet in each client. All clients only belong to one environment and
    do not move between them.
    - We have a single Puppet configuration to manage all environments.
    Various conditional statements based on environment, application type,
    hostname, etc. control what each client receives for its
    configuration.
    - There are separate servers for each environment for security reasons
    (primarily sensitive information that can only exist in the production
    environment).
    - The Puppet configuration maintained via a Git repo, currently on a
    single branch.
    - Each person on the admin team checks out own copy of the repo, make
    changes, commits the changes, then updates each environment on the
    Puppet servers for the changes to take effect.

    There are several issues with this process, unfortunately:

    - Every so often a configuration mistake will adversely affect an
    entire environment, and much of the time is only noticed _after_ the
    changes are pushed out. As a result, local changes tend to be made in
    the development environment for testing and sometimes aren't committed
    for a long time, leaving discrepancies between the environments which
    can lead to other subtle issues.
    - Less frequent but still occuring often enough, changes can still
    have subtle issues which cause things to work in one environment and
    break horribly in another; this is especially bad when the broken
    environment is the production one.
    - The configuration for a given type of client is complex enough that
    to change a client to a different application type (what we primarily
    key most of our configuration off of, followed by the environment) to
    test against a server would require rebuilding the client, which is a
    25-45 minute process; too slow for simple changes and even too slow
    for all but the most complex changes, given how many changes we make
    in a single day.
    - We allow our users to create local VMs that the development Puppet
    server can key off their names to create a given configuration, but
    since the configuration for the various environments is shared in a
    Puppet configuration, potential for users point their puppet agents to
    the production environment is a concern (due to the sensitive
    information there).

    After discussion with a few coworkers, the following process was laid
    out to try to implement to resolve these issues:

    - Create separate branches for each of the environments and have only
    the matching branch checked out on the primary Puppet servers; changes
    will be merged into the various branches one at a time to prevent
    unintentional changes in a given environment before testing can be
    done on that environment.
    - Ensure a client in a given environment can ONLY run against that
    configuration (e.g. disallow a client in the development environment
    requesting the production configuration).
    - Each person on the admin team will have a test server where they can
    create their own branches from the Git repo for the changes they're
    working and use their test server to test changes against existing
    clients in the various environments (preventing the need to build out
    a new test client(s) to validate each change). The existing clients
    would only be run in no-op mode against the test servers.

    The reason for each person on the admin team to have their own test
    server that has access to all the environments is considered since:

    - Having a different server for each environment would be affected by
    the tight hardware resources currently.
    - The need for having separate test servers would prevent needing to
    use the primary servers for testing, which is difficult due to
    multiple admins continuously making changes and needing to test them
    without disturbing the other admins' work, along with not affecting
    the current primary servers from being able to properly handle their
    existing clients.

    What this all boils down to is I'm trying to find a way to deal with a
    single test server trying to be able to communicate with existing
    clients in all the environments; most of the current configuration
    would work fine except for the cert issue, which is the sticking point
    at this time. Any solution on how to handle this in the most
    straightforward manner would greatly be appreciated, as my research
    has been leading to solutions far more complex than what I would like
    (such as load balancing for the CA or trying to synchronize the certs
    across the various systems).

    Hopefully this made sense and I can find someone who can give advice
    on how to proceed with this issue. Thanks in advance.

    - Ken Lareau
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/HvWvZVO4JbYJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.


    --
    - Ken Lareau

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Trevor Smith at Apr 18, 2012 at 7:29 pm
    I'll take a stab at some of this. Hopefully I'm correctly understanding
    your issue.

    Am I correct in the following? :

    You define 3 environments development, staging, and production. These
    environments are defined as such in Puppet but they are also separate
    environments within your network, for the sake of clarity I'll call them
    zones from here out?

    Each zone has a Puppet Master server.

    Each Puppet Master server has three environments defined development,
    staging, and production. Each environment has the full git repository with
    the applicable branch checked out.

    The clients in each zone connect to the Puppet Master in their zone and
    pull their configs from the corresponding environment. So a
    staging_zone_client connects to staging_zone_master and pulls from the
    staging_environment.

    If that's correct then:

    You already have three separate Puppet Masters so the environments are
    redundant. As configured staging_zone_client can pull from
    production_environment using --environment. One fix could be to define
    only the production environment on each zone's Puppet Master and check out
    the applicable branch in only the production environment. As long as you
    never check out the production branch in development or staging then
    clients from those zones couldn't pull the settings for production zone as
    it's just not available. As long as they cannot connect to the other
    zone's Puppet Master, preventable by network segmentation, certs etc...

    Within each zone you could then define environments such as development and
    testing for conducting those activities within each zone. So you'd have
    staging/dev and staging/test branches checked out in those environments. I
    guess you could extend that and create environments for each admin within
    each zone that would allow the admin to use the --environment option for
    clients to test their work within a zone. This would result in a lot of
    environments, and probably a lot of branches, but you wouldn't need a test
    Puppet Master for each admin.

    I'd think this would introduce the problem of making it difficult to reuse
    modules between zones as I'd think you'd end up basically managing three
    completely different branches. Unless the sensitive data you're worried
    about is not being stored in your puppet repo and you have no issues
    merging changes made to the production branch into the development and
    testing branches, plus your admins will have a lot of different topic
    branches to deal with. Long run you'd probably want to move zone specific
    settings out of your modules and use something like hiera so you can
    standardize your modules across zones and just pull in the location
    settings using hiera.

    Hope I understood your problems correctly and this is helpful..
    On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote:

    Hello folks,

    After some conversation on #puppet on Freenode IRC, Eric Sorenson
    requested I repost the information and question here, so I am doing so
    and hopefully it will all make sense...

    We currently have a well-established and relatively complex Puppet
    setup in place at my company and I'm in the process of trying to
    streamline changes as well as implement better testing to ensure
    minimal disruption or issues when making those changes. Some
    information on the current situation:

    - There are currently three environments: development, staging,
    production. These are controlled via the '--environment' setting for
    puppet in each client. All clients only belong to one environment and
    do not move between them.
    - We have a single Puppet configuration to manage all environments.
    Various conditional statements based on environment, application type,
    hostname, etc. control what each client receives for its
    configuration.
    - There are separate servers for each environment for security reasons
    (primarily sensitive information that can only exist in the production
    environment).
    - The Puppet configuration maintained via a Git repo, currently on a
    single branch.
    - Each person on the admin team checks out own copy of the repo, make
    changes, commits the changes, then updates each environment on the
    Puppet servers for the changes to take effect.

    There are several issues with this process, unfortunately:

    - Every so often a configuration mistake will adversely affect an
    entire environment, and much of the time is only noticed _after_ the
    changes are pushed out. As a result, local changes tend to be made in
    the development environment for testing and sometimes aren't committed
    for a long time, leaving discrepancies between the environments which
    can lead to other subtle issues.
    - Less frequent but still occuring often enough, changes can still
    have subtle issues which cause things to work in one environment and
    break horribly in another; this is especially bad when the broken
    environment is the production one.
    - The configuration for a given type of client is complex enough that
    to change a client to a different application type (what we primarily
    key most of our configuration off of, followed by the environment) to
    test against a server would require rebuilding the client, which is a
    25-45 minute process; too slow for simple changes and even too slow
    for all but the most complex changes, given how many changes we make
    in a single day.
    - We allow our users to create local VMs that the development Puppet
    server can key off their names to create a given configuration, but
    since the configuration for the various environments is shared in a
    Puppet configuration, potential for users point their puppet agents to
    the production environment is a concern (due to the sensitive
    information there).

    After discussion with a few coworkers, the following process was laid
    out to try to implement to resolve these issues:

    - Create separate branches for each of the environments and have only
    the matching branch checked out on the primary Puppet servers; changes
    will be merged into the various branches one at a time to prevent
    unintentional changes in a given environment before testing can be
    done on that environment.
    - Ensure a client in a given environment can ONLY run against that
    configuration (e.g. disallow a client in the development environment
    requesting the production configuration).
    - Each person on the admin team will have a test server where they can
    create their own branches from the Git repo for the changes they're
    working and use their test server to test changes against existing
    clients in the various environments (preventing the need to build out
    a new test client(s) to validate each change). The existing clients
    would only be run in no-op mode against the test servers.

    The reason for each person on the admin team to have their own test
    server that has access to all the environments is considered since:

    - Having a different server for each environment would be affected by
    the tight hardware resources currently.
    - The need for having separate test servers would prevent needing to
    use the primary servers for testing, which is difficult due to
    multiple admins continuously making changes and needing to test them
    without disturbing the other admins' work, along with not affecting
    the current primary servers from being able to properly handle their
    existing clients.

    What this all boils down to is I'm trying to find a way to deal with a
    single test server trying to be able to communicate with existing
    clients in all the environments; most of the current configuration
    would work fine except for the cert issue, which is the sticking point
    at this time. Any solution on how to handle this in the most
    straightforward manner would greatly be appreciated, as my research
    has been leading to solutions far more complex than what I would like
    (such as load balancing for the CA or trying to synchronize the certs
    across the various systems).

    Hopefully this made sense and I can find someone who can give advice
    on how to proceed with this issue. Thanks in advance.

    - Ken Lareau
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/2BDCbntAOXoJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ken Lareau at Apr 18, 2012 at 8:14 pm
    Trevor,

    Thank you for the response; I believe you got the idea pretty well and
    while your suggestion makes sense, it is something we definitely can't
    follow through with right now; our configuration is massive and complex and
    having to maintain three different yet similar sets of configuration would
    be difficult and reduce our response time to necessary user changes (of
    which we get anywhere from 5-10 a day). It's just not feasible without a
    complete reworking of how we do things right now, and not at the top of our
    priority lists.

    I do appreciate the input, however. Thank you.

    - Ken Lareau

    On Wed, Apr 18, 2012 at 12:28 PM, Trevor Smith wrote:

    I'll take a stab at some of this. Hopefully I'm correctly understanding
    your issue.

    Am I correct in the following? :

    You define 3 environments development, staging, and production. These
    environments are defined as such in Puppet but they are also separate
    environments within your network, for the sake of clarity I'll call them
    zones from here out?

    Each zone has a Puppet Master server.

    Each Puppet Master server has three environments defined development,
    staging, and production. Each environment has the full git repository with
    the applicable branch checked out.

    The clients in each zone connect to the Puppet Master in their zone and
    pull their configs from the corresponding environment. So a
    staging_zone_client connects to staging_zone_master and pulls from the
    staging_environment.

    If that's correct then:

    You already have three separate Puppet Masters so the environments are
    redundant. As configured staging_zone_client can pull from
    production_environment using --environment. One fix could be to define
    only the production environment on each zone's Puppet Master and check out
    the applicable branch in only the production environment. As long as you
    never check out the production branch in development or staging then
    clients from those zones couldn't pull the settings for production zone as
    it's just not available. As long as they cannot connect to the other
    zone's Puppet Master, preventable by network segmentation, certs etc...

    Within each zone you could then define environments such as development
    and testing for conducting those activities within each zone. So you'd
    have staging/dev and staging/test branches checked out in those
    environments. I guess you could extend that and create environments for
    each admin within each zone that would allow the admin to use the
    --environment option for clients to test their work within a zone. This
    would result in a lot of environments, and probably a lot of branches, but
    you wouldn't need a test Puppet Master for each admin.

    I'd think this would introduce the problem of making it difficult to reuse
    modules between zones as I'd think you'd end up basically managing three
    completely different branches. Unless the sensitive data you're worried
    about is not being stored in your puppet repo and you have no issues
    merging changes made to the production branch into the development and
    testing branches, plus your admins will have a lot of different topic
    branches to deal with. Long run you'd probably want to move zone specific
    settings out of your modules and use something like hiera so you can
    standardize your modules across zones and just pull in the location
    settings using hiera.

    Hope I understood your problems correctly and this is helpful..
    On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote:

    Hello folks,

    After some conversation on #puppet on Freenode IRC, Eric Sorenson
    requested I repost the information and question here, so I am doing so
    and hopefully it will all make sense...

    We currently have a well-established and relatively complex Puppet
    setup in place at my company and I'm in the process of trying to
    streamline changes as well as implement better testing to ensure
    minimal disruption or issues when making those changes. Some
    information on the current situation:

    - There are currently three environments: development, staging,
    production. These are controlled via the '--environment' setting for
    puppet in each client. All clients only belong to one environment and
    do not move between them.
    - We have a single Puppet configuration to manage all environments.
    Various conditional statements based on environment, application type,
    hostname, etc. control what each client receives for its
    configuration.
    - There are separate servers for each environment for security reasons
    (primarily sensitive information that can only exist in the production
    environment).
    - The Puppet configuration maintained via a Git repo, currently on a
    single branch.
    - Each person on the admin team checks out own copy of the repo, make
    changes, commits the changes, then updates each environment on the
    Puppet servers for the changes to take effect.

    There are several issues with this process, unfortunately:

    - Every so often a configuration mistake will adversely affect an
    entire environment, and much of the time is only noticed _after_ the
    changes are pushed out. As a result, local changes tend to be made in
    the development environment for testing and sometimes aren't committed
    for a long time, leaving discrepancies between the environments which
    can lead to other subtle issues.
    - Less frequent but still occuring often enough, changes can still
    have subtle issues which cause things to work in one environment and
    break horribly in another; this is especially bad when the broken
    environment is the production one.
    - The configuration for a given type of client is complex enough that
    to change a client to a different application type (what we primarily
    key most of our configuration off of, followed by the environment) to
    test against a server would require rebuilding the client, which is a
    25-45 minute process; too slow for simple changes and even too slow
    for all but the most complex changes, given how many changes we make
    in a single day.
    - We allow our users to create local VMs that the development Puppet
    server can key off their names to create a given configuration, but
    since the configuration for the various environments is shared in a
    Puppet configuration, potential for users point their puppet agents to
    the production environment is a concern (due to the sensitive
    information there).

    After discussion with a few coworkers, the following process was laid
    out to try to implement to resolve these issues:

    - Create separate branches for each of the environments and have only
    the matching branch checked out on the primary Puppet servers; changes
    will be merged into the various branches one at a time to prevent
    unintentional changes in a given environment before testing can be
    done on that environment.
    - Ensure a client in a given environment can ONLY run against that
    configuration (e.g. disallow a client in the development environment
    requesting the production configuration).
    - Each person on the admin team will have a test server where they can
    create their own branches from the Git repo for the changes they're
    working and use their test server to test changes against existing
    clients in the various environments (preventing the need to build out
    a new test client(s) to validate each change). The existing clients
    would only be run in no-op mode against the test servers.

    The reason for each person on the admin team to have their own test
    server that has access to all the environments is considered since:

    - Having a different server for each environment would be affected by
    the tight hardware resources currently.
    - The need for having separate test servers would prevent needing to
    use the primary servers for testing, which is difficult due to
    multiple admins continuously making changes and needing to test them
    without disturbing the other admins' work, along with not affecting
    the current primary servers from being able to properly handle their
    existing clients.

    What this all boils down to is I'm trying to find a way to deal with a
    single test server trying to be able to communicate with existing
    clients in all the environments; most of the current configuration
    would work fine except for the cert issue, which is the sticking point
    at this time. Any solution on how to handle this in the most
    straightforward manner would greatly be appreciated, as my research
    has been leading to solutions far more complex than what I would like
    (such as load balancing for the CA or trying to synchronize the certs
    across the various systems).

    Hopefully this made sense and I can find someone who can give advice
    on how to proceed with this issue. Thanks in advance.

    - Ken Lareau
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/2BDCbntAOXoJ.

    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.


    --
    - Ken Lareau

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Luke Bigum at Apr 19, 2012 at 8:31 am
    Hi Ken,

    Your site and your practices (and the problems that go with it) are
    similar to our own. I've answered by describing what we think the way
    forward for Puppet is for us, hopefully you find it helpful and get some
    ideas :-)

    I feel your pain when you say there are changes languishing in Dev for
    too long that get inadvertently pushed out to other environments. Once I
    accidentally rolled out a whole set of 'work in progress' changes that a
    colleague of mine was committing and pushing to one environment (we use
    Git) and I cut a new tag and pushed a whole heap of these unfinished
    changes to the next environment up the chain.

    After this I realised our current model really wasn't going to work in
    the long run: this happened with only two Admins working on Puppet in
    the last week when we employ almost a dozen who could potentially make
    the same mistakes. While we're still stuck in this model now, we think
    the way forward for us is to take a page out of our Development Team's
    book and set up a fully fledged CI environment for Puppet. It would be
    even more awesome if our Developer's CI was also our CI - but
    considering they've got half a million unit tests and we've got zero,
    we've got a long way to go.

    How this plays out practically is that all our admins commit into our
    Puppet repository and those changes start getting applied immediately to
    our Dev/CI servers and they start getting tested against immediately
    (hence the unit tests). Problems will always occur, but they will occur
    almost immediately and we'll have feedback to fix it.

    In response to your reply to Trevor:

    Can you explain how you see maintaining three slightly different
    environments with one Puppet Master and a firewall is more difficult
    than three individual Puppet Masters? You've already got all the
    differences written down in your existing manifests - different IP
    addresses for different services, different users/passwords, etc, maybe
    slightly different packages, or am I missing something?

    Without knowing more about your site (so hopefully this suggestion is
    relevant), here's what I would do: I would slowly start to encourage
    people to separate out your data from your code using a shared
    repository. Take a look at Hiera if you haven't already, I'd recommend
    using it but other data sources would suffice as well.

    So right now you've got your Dev data in your Dev modules, your Staging
    data in your Staging modules, etc. Take one of your modules (lets say
    DNS, it's easy), and put it's data (DNS server IP addresses) into a
    shared data store that each of your Dev, Staging and Prod Puppet Masters
    can access. Then change your Dev module to remove the data and pull it
    from the data store. Then do Staging, the same data store but different
    information, then finally Prod. If your three separate DNS modules don't
    look *exactly* the same now, you've done something wrong. Do that for
    every module and you've now got no reason not to run a single Puppet
    Master. The changes would take some time to do every module but if you
    encourage every admin to make a small step forward each change they
    make, eventually you'd get there.

    What kind of changes are you making 5-10 times a day? The same changes
    for the same people? We have a situation here where our Developers
    sometimes need to change the configuration of our application every
    hour. Right now we don't manage those systems with Puppet because making
    that many changes of the same thing over and over is a monumental waste
    of our time.

    I hack on Hiera a little bit to get some extra functionality from it,
    mainly the ability to tier my data sources. What I plan on doing is to
    have a low priority data store that's managed by Developers. Since all
    our modules will (eventually) look to the data stores to configure
    themselves, our Developers will be able to make their own system
    configuration changes. Our Admins control the modules (we write *how*
    it's done) and the higher priority data stores so we can override
    whatever the Developers enter if need be (we don't want a Developer
    changing the root password now). The Developers then trigger their own
    Puppet run and they've reconfigured the system using the same method
    (same Puppet modules) as Production so I don't have to worry about
    things "being done differently in Dev".

    -Luke
    On 18/04/12 21:08, Ken Lareau wrote:
    Trevor,

    Thank you for the response; I believe you got the idea pretty well and
    while your suggestion makes sense, it is something we definitely can't
    follow through with right now; our configuration is massive and
    complex and having to maintain three different yet similar sets of
    configuration would be difficult and reduce our response time to
    necessary user changes (of which we get anywhere from 5-10 a day).
    It's just not feasible without a complete reworking of how we do
    things right now, and not at the top of our priority lists.

    I do appreciate the input, however. Thank you.

    - Ken Lareau


    On Wed, Apr 18, 2012 at 12:28 PM, Trevor Smith
    wrote:

    I'll take a stab at some of this. Hopefully I'm correctly
    understanding your issue.

    Am I correct in the following? :

    You define 3 environments development, staging, and production.
    These environments are defined as such in Puppet but they are
    also separate environments within your network, for the sake of
    clarity I'll call them zones from here out?

    Each zone has a Puppet Master server.

    Each Puppet Master server has three environments defined
    development, staging, and production. Each environment has the
    full git repository with the applicable branch checked out.

    The clients in each zone connect to the Puppet Master in their
    zone and pull their configs from the corresponding environment.
    So a staging_zone_client connects to staging_zone_master and
    pulls from the staging_environment.

    If that's correct then:

    You already have three separate Puppet Masters so the environments
    are redundant. As configured staging_zone_client can pull from
    production_environment using --environment. One fix could be to
    define only the production environment on each zone's Puppet
    Master and check out the applicable branch in only the production
    environment. As long as you never check out the production branch
    in development or staging then clients from those zones couldn't
    pull the settings for production zone as it's just not available.
    As long as they cannot connect to the other zone's Puppet Master,
    preventable by network segmentation, certs etc...

    Within each zone you could then define environments such as
    development and testing for conducting those activities within
    each zone. So you'd have staging/dev and staging/test branches
    checked out in those environments. I guess you could extend that
    and create environments for each admin within each zone that would
    allow the admin to use the --environment option for clients to
    test their work within a zone. This would result in a lot of
    environments, and probably a lot of branches, but you wouldn't
    need a test Puppet Master for each admin.

    I'd think this would introduce the problem of making it difficult
    to reuse modules between zones as I'd think you'd end up basically
    managing three completely different branches. Unless the
    sensitive data you're worried about is not being stored in your
    puppet repo and you have no issues merging changes made to the
    production branch into the development and testing branches, plus
    your admins will have a lot of different topic branches to deal
    with. Long run you'd probably want to move zone specific settings
    out of your modules and use something like hiera so you can
    standardize your modules across zones and just pull in the
    location settings using hiera.

    Hope I understood your problems correctly and this is helpful..

    On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote:

    Hello folks,

    After some conversation on #puppet on Freenode IRC, Eric Sorenson
    requested I repost the information and question here, so I am
    doing so
    and hopefully it will all make sense...

    We currently have a well-established and relatively complex
    Puppet
    setup in place at my company and I'm in the process of trying to
    streamline changes as well as implement better testing to ensure
    minimal disruption or issues when making those changes. Some
    information on the current situation:

    - There are currently three environments: development, staging,
    production. These are controlled via the '--environment'
    setting for
    puppet in each client. All clients only belong to one
    environment and
    do not move between them.
    - We have a single Puppet configuration to manage all
    environments.
    Various conditional statements based on environment,
    application type,
    hostname, etc. control what each client receives for its
    configuration.
    - There are separate servers for each environment for security
    reasons
    (primarily sensitive information that can only exist in the
    production
    environment).
    - The Puppet configuration maintained via a Git repo,
    currently on a
    single branch.
    - Each person on the admin team checks out own copy of the
    repo, make
    changes, commits the changes, then updates each environment on
    the
    Puppet servers for the changes to take effect.

    There are several issues with this process, unfortunately:

    - Every so often a configuration mistake will adversely affect an
    entire environment, and much of the time is only noticed
    _after_ the
    changes are pushed out. As a result, local changes tend to be
    made in
    the development environment for testing and sometimes aren't
    committed
    for a long time, leaving discrepancies between the
    environments which
    can lead to other subtle issues.
    - Less frequent but still occuring often enough, changes can
    still
    have subtle issues which cause things to work in one
    environment and
    break horribly in another; this is especially bad when the broken
    environment is the production one.
    - The configuration for a given type of client is complex
    enough that
    to change a client to a different application type (what we
    primarily
    key most of our configuration off of, followed by the
    environment) to
    test against a server would require rebuilding the client,
    which is a
    25-45 minute process; too slow for simple changes and even too
    slow
    for all but the most complex changes, given how many changes
    we make
    in a single day.
    - We allow our users to create local VMs that the development
    Puppet
    server can key off their names to create a given
    configuration, but
    since the configuration for the various environments is shared
    in a
    Puppet configuration, potential for users point their puppet
    agents to
    the production environment is a concern (due to the sensitive
    information there).

    After discussion with a few coworkers, the following process
    was laid
    out to try to implement to resolve these issues:

    - Create separate branches for each of the environments and
    have only
    the matching branch checked out on the primary Puppet servers;
    changes
    will be merged into the various branches one at a time to prevent
    unintentional changes in a given environment before testing
    can be
    done on that environment.
    - Ensure a client in a given environment can ONLY run against
    that
    configuration (e.g. disallow a client in the development
    environment
    requesting the production configuration).
    - Each person on the admin team will have a test server where
    they can
    create their own branches from the Git repo for the changes
    they're
    working and use their test server to test changes against
    existing
    clients in the various environments (preventing the need to
    build out
    a new test client(s) to validate each change). The existing
    clients
    would only be run in no-op mode against the test servers.

    The reason for each person on the admin team to have their own
    test
    server that has access to all the environments is considered
    since:

    - Having a different server for each environment would be
    affected by
    the tight hardware resources currently.
    - The need for having separate test servers would prevent
    needing to
    use the primary servers for testing, which is difficult due to
    multiple admins continuously making changes and needing to
    test them
    without disturbing the other admins' work, along with not
    affecting
    the current primary servers from being able to properly handle
    their
    existing clients.

    What this all boils down to is I'm trying to find a way to
    deal with a
    single test server trying to be able to communicate with existing
    clients in all the environments; most of the current
    configuration
    would work fine except for the cert issue, which is the
    sticking point
    at this time. Any solution on how to handle this in the most
    straightforward manner would greatly be appreciated, as my
    research
    has been leading to solutions far more complex than what I
    would like
    (such as load balancing for the CA or trying to synchronize
    the certs
    across the various systems).

    Hopefully this made sense and I can find someone who can give
    advice
    on how to proceed with this issue. Thanks in advance.

    - Ken Lareau

    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/2BDCbntAOXoJ.

    To post to this group, send email to puppet-users@googlegroups.com
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.




    --
    - Ken Lareau


    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.

    --
    Luke Bigum

    Information Systems
    Ph: +44 (0) 20 3192 2520
    luke.bigum@lmax.com | http://www.lmax.com
    LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN



    FX and CFDs are leveraged products that can result in losses exceeding
    your deposit. They are not suitable for everyone so please ensure you
    fully understand the risks involved. The information in this email is not
    directed at residents of the United States of America or any other
    jurisdiction where trading in CFDs and/or FX is restricted or prohibited
    by local laws or regulations.

    The information in this email and any attachment is confidential and is
    intended only for the named recipient(s). The email may not be disclosed
    or used by any person other than the addressee, nor may it be copied in
    any way. If you are not the intended recipient please notify the sender
    immediately and delete any copies of this message. Any unauthorised
    copying, disclosure or distribution of the material in this e-mail is
    strictly forbidden.

    LMAX operates a multilateral trading facility. Authorised and regulated
    by the Financial Services Authority (firm registration number 509778) and
    is registered in England and Wales (number 06505809).
    Our registered address is Yellow Building, 1A Nicholas Road, London, W11
    4AN.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedApr 18, '12 at 7:12a
activeApr 19, '12 at 8:31a
posts6
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase