After some conversation on #puppet on Freenode IRC, Eric Sorenson
requested I repost the information and question here, so I am doing so
and hopefully it will all make sense...
We currently have a well-established and relatively complex Puppet
setup in place at my company and I'm in the process of trying to
streamline changes as well as implement better testing to ensure
minimal disruption or issues when making those changes. Some
information on the current situation:
- There are currently three environments: development, staging,
production. These are controlled via the '--environment' setting for
puppet in each client. All clients only belong to one environment and
do not move between them.
- We have a single Puppet configuration to manage all environments.
Various conditional statements based on environment, application type,
hostname, etc. control what each client receives for its
- There are separate servers for each environment for security reasons
(primarily sensitive information that can only exist in the production
- The Puppet configuration maintained via a Git repo, currently on a
- Each person on the admin team checks out own copy of the repo, make
changes, commits the changes, then updates each environment on the
Puppet servers for the changes to take effect.
There are several issues with this process, unfortunately:
- Every so often a configuration mistake will adversely affect an
entire environment, and much of the time is only noticed _after_ the
changes are pushed out. As a result, local changes tend to be made in
the development environment for testing and sometimes aren't committed
for a long time, leaving discrepancies between the environments which
can lead to other subtle issues.
- Less frequent but still occuring often enough, changes can still
have subtle issues which cause things to work in one environment and
break horribly in another; this is especially bad when the broken
environment is the production one.
- The configuration for a given type of client is complex enough that
to change a client to a different application type (what we primarily
key most of our configuration off of, followed by the environment) to
test against a server would require rebuilding the client, which is a
25-45 minute process; too slow for simple changes and even too slow
for all but the most complex changes, given how many changes we make
in a single day.
- We allow our users to create local VMs that the development Puppet
server can key off their names to create a given configuration, but
since the configuration for the various environments is shared in a
Puppet configuration, potential for users point their puppet agents to
the production environment is a concern (due to the sensitive
After discussion with a few coworkers, the following process was laid
out to try to implement to resolve these issues:
- Create separate branches for each of the environments and have only
the matching branch checked out on the primary Puppet servers; changes
will be merged into the various branches one at a time to prevent
unintentional changes in a given environment before testing can be
done on that environment.
- Ensure a client in a given environment can ONLY run against that
configuration (e.g. disallow a client in the development environment
requesting the production configuration).
- Each person on the admin team will have a test server where they can
create their own branches from the Git repo for the changes they're
working and use their test server to test changes against existing
clients in the various environments (preventing the need to build out
a new test client(s) to validate each change). The existing clients
would only be run in no-op mode against the test servers.
The reason for each person on the admin team to have their own test
server that has access to all the environments is considered since:
- Having a different server for each environment would be affected by
the tight hardware resources currently.
- The need for having separate test servers would prevent needing to
use the primary servers for testing, which is difficult due to
multiple admins continuously making changes and needing to test them
without disturbing the other admins' work, along with not affecting
the current primary servers from being able to properly handle their
What this all boils down to is I'm trying to find a way to deal with a
single test server trying to be able to communicate with existing
clients in all the environments; most of the current configuration
would work fine except for the cert issue, which is the sticking point
at this time. Any solution on how to handle this in the most
straightforward manner would greatly be appreciated, as my research
has been leading to solutions far more complex than what I would like
(such as load balancing for the CA or trying to synchronize the certs
across the various systems).
Hopefully this made sense and I can find someone who can give advice
on how to proceed with this issue. Thanks in advance.
- Ken Lareau
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to email@example.com.
To unsubscribe from this group, send email to firstname.lastname@example.org.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.