[Puppet Users] a complete solution for puppet

Sounds like you should be talking to your managers about buying Puppet Enterprise.

I see. so it is on purpose to make it not easy to use so the
enterprise can be sold? :)

On Wed, Jul 25, 2012 at 2:02 PM, Christopher Wood
wrote:

There are different skill levels at different tasks in the enterprise space, and it is legitimate that some organizations are better off with a prefabbed installer for a configuration management system.

I've created a puppet installation of reasonable complexity without puppet enterprise, but that is possibly just me:

$ cd files/puppet/svn/prod/trunk
$ ls manifests/nodes | wc -l
43
$ find modules -name "*pp" | wc -l
174

That's not to say I don't salivate a bit at the thought of Puppet Enterprise, but my budget of $0 doesn't help there. Or perhaps a career-long $0 budget has helped, in that I'm more used to building from components instead of buying the package. People who are more used to buying than building may be better off with a different situation than mine.

Hey, Chris: so that begs the question, do you think you have some "secret"
or are just
happier with fewer flashy gui's, more install/deployment scripts, and so
forth.

In other words, do you think the scaling of Open Puppet is adequate to
scale much larger
without the flash?

Or, is there something fundamentally holding back Open Puppet from handling
thousands, tens of thousands, or hundreds of thousands of nodes, in your
opinion?

Cheers,
Stuart

I have heard from various other puppet users of Open Puppet
installations of tens of thousands of nodes (at least last I heard,
there's probably people with more) so it definitely scales that high.
Generally they've had to restrict themselves on some features or
sometimes use Puppet in slightly strange ways to improve scaling, but
people are definitely doing it today.

The flash you get from PE just makes things easier. Most of us
started with a fairly small Puppet trial and over time developed
modules to build up an entire puppet infrastructure. I have a fairly
(badly) written module that installs puppetmasters, sets up repos,
hiera, databases, everything needed to fully deploy a relatively
scalable Puppet infrastructure. I think there's a lot of us out
there, maybe a few of us should get together, share what we have, and
try to build a combined module that people can use to help bootstrap
and scale.

But then, it's not just puppet that people are scaling. If somebody thinks that they're going to point 100k nodes at a single virtual machine running puppet and have everything work at 99.999% uptime, they're making a mistake. (The same mistake that we've all seen for mail, radius, dns, et cetera.) Even if that works, are they willing to lose a cluster's configuration management if a single VM goes down? So now we have multiple puppet servers. Unix-like directories get slow when we add hundreds of thousands of directory entries. Let's stop tossing our node definitions in a single directory. How do we keep the certificates in sync? Now we have a system to sync certs. Can our switches handle the load of all that network traffic? Let's make sure we have redundant switches in our network core. Do we really want every server to depend on one set of puppetmasters? Let's break things out into pods. Can't keep the pods in sync? Maybe centralized is the way to go. Your whole datacentre checks into the puppetmaster at the same time every hour? It's time to spread thousands of requests out over the 3600 seconds you have in each hour, or add more backend puppetmasters or check in less often. There's a ton more of these scaling items.

In short: scaling puppet is about more than puppet. The puppet component is ready to compile a catalog from your manifests and send it to the node, yes. Every other layer has to be ready to scale up in support of that goal.

Best.....Post.....Ever

On Fri, Jul 27, 2012 at 10:58 AM, Christopher Wood
wrote:

+1

Seconded. Scaling is not a stand-alone problem. Indeed beyond the
infrastructure components you also need to consider how you scale risk and
availability across multiple combinations of architecture and stack, your
Puppet configuration, your module style, ENC/Hiera design and how you
handle commonalities and exceptions. The "default" setup and configuration
for any application or stack generally has a sweet spot. And generally that
sweet spot isn't at 1 machine or 100,000 machines but tries to strike a
balance in the middle. You will almost always need to tune the application
or stack to your sweet spot.

Regards

James

I decided to go with Puppet Enterprise and will be upgrading this weekend or possibly today.

--Stuart

Via Apple iPhone 4S on the AT&T Wireless Network

If you don't know how to migrate puppet to passenger (either on nginx or
apache), then I don't understand how come that you are able to configure
your machines through puppet manifests?!? It ain't that hard.

You could try using foreman installer(aka as plain puppet modules)[1],
which does take care for the following

1. apache configuration + passenger
2. puppet master (with optional dynamic git branches / environments)
3. foreman stuff

if you don't need storeconfigs, then you don't have to use it.

if you need to scale to more than one master, my guess is that the main
thing to overcome is the CA setup (single CA server etc) and thats the kind
of knowledge you need to have regardless of an installer.

Ohad

[1] https://github.com/theforeman/foreman-installer#readme

You probably want Puppet Enterprise as it comes with one installer for
all these dependencies

http://puppetlabs.com/puppet/faq/

-Stefan

Is it a bundle with everything prepackaged? Or does it use system's
httpd for example?

I hate bundles :D

I'm sure that the designers of puppet made every effort to make puppet as usable and scalable as they could when it was being designed. This is proven time and time again but the sites that currently have puppet deployed with tens of thousands of nodes. You should take care when making such bold statements, but I assume of course that it was not you intention to insult anyone.

We originally rolled puppet out with 10 machines. Managing 10 machines without puppet was 10 times more work than managing 10 machines with puppet. We had to log into each host and apply updates as appropriately. We had to develop management scripts to manage them as well. Puppet helped unify this and ensure less drift between machines so any more than one machine and puppet can be really handy.

Now the other thing is that much of Puppet's configuration options can be swapped out for better scalability. For example the default storage backend could be MySQL, PostgreSQL or PuppetDB. The built in webrick server swapped for {Apache,cherokee,nginx}+passenger. The management from CLI to something like Puppet Dashboard, The Foreman or the Puppet Enterprise solution. It's completely up to you the route you take.

So does puppet scale well for large environments by default no, but that's ok, swap out a component that's the bottleneck and move on. It seems that your complaint is that there isn't a single "all-in-one" solution for you to choose from, when in fact there is, Puppet Enterprise. This comes with all the tools you'd need to scale puppet in a "black box" style. If you're too cheap and don't want to pay for that you can try The Foreman installer which does a pretty good job of installing the toolchain to scale puppet to larger environments using Apache and Passenger and you don't really need to know squat about how the components go together. You could find a "How To" or tutorial online and just follow it line by line if you want to! Go for it!

At any rate, you're going to have to learn about the toolchain associated with scaling puppet anyway. You're going to need to learn how to monitor and tweak Apache and Passenger if you need to scale the web components. You're going to need to know how to tweak MySQL, PostgreSQL or Puppet DB for stored configs. For PuppetDB you're also going to have to learn a bit about tuning Java to make it scale too!

Learn the tools or pay for a tuned black box. You can't have it both ways. If you want a rather simple way to deploy Puppet+Apache+Passenger+The Foreman, use The Foreman installer which works quite well. Have fun!