FAQ
I am trying to establish what looks like a common pattern for scaling
puppet. My main departure is that I'm using an F5 rather than an apache
load balancer. Namely, I want to have my puppet agents go through the F5
to a pool of "master only" systems, and any Certificate activity to get
proxied by those masters through to one single Certificate Authority. That
CA system is not part of the F5 pool, it's role is to provide CA, Puppetdb
and Postgresql. It is configured as a master because that was the easiest
way to get a CA stood up, but I don't intend to use it as a master in
normal operation (and in fact I don't plan to have it hosting any modules).

I'm using RHEL 6, Apache, and Passenger, and Open Source Puppet.

I initially set up passenger using puppetlabs/passenger from the Forge,
(which got me most of the way there but not fully configured). All of
these steps worked fine for the CA system to configure it as a working
master (I have tested by registering systems with it, but then done puppet
cert clean and wiped the test systems' ssl directories).

I then set up my first master-only system the same way, except I didn't
actually start the master service (as the docs say) until after I had set
ca = false and ca_server = $MY_CA_SERVER into /etc/puppet.conf. I also
made the necessary changes listed at
http://docs.puppetlabs.com/guides/scaling_multiple_masters.html, including
the certificate access on the CA system, the SSLProxyEngine on and
ProxyPassMatch lines in the VHost definition in
/etc/httpd/conf.d/puppetmaster.conf. I'm positive I followed all the steps
in the docs in order, but I'm not having any luck with external agents.

If I run puppet agent -t on the master-only system (with it's "server" in
puppet.conf set to itself) it works fine--it can talk to the CA and talk to
itself, and all is right with the world.

If I run puppet agent -t on a client host, pointing at the load balancer's
address (or even pointing direclty at the master-only system's real
hostname), I get:

[root@elmer ~]# puppet agent -t
Info: Creating a new SSL key for elmer.allstate.com
Error: Could not request certificate: Error 400 on SERVER: this master is
not a CA
Exiting; failed to retrieve certificate and waitforcert is disabled


I've looked at the logs, enabled debug logging in the webserver with
LogLevel, dug around everywhere I can think of, and I see no sign of any
actual proxying going on. tcpdump certainly shows no attempt by the
master-only system to contact the CA.

What it LOOKS like is happening is that apache is not actually proxying
anything, the request gets passed to the puppet master app running under
passenger, and it (rightly) says "I'm not a CA" because
/etc/puppet/puppet.conf says so.

I do not see any errors in the logs about proxy attempts failing for this
agent. I do see workers being attached for proxy purposes:

[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1833): proxy: grabbed
scoreboard slot 0 in child 27434 for worker
https://caserver.allstate.com:8140/$1
[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1852): proxy: worker
https://caserver.allstate.com:8140/$1 already initialized
[Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1949): proxy: initialized
single connection worker 0 in child 27434 for caserver.allstate.com)


I've repeatedly re-checked the settings in /etc/puppet.conf
/etc/httpd/conf.d/passenger.conf, /et/chttpd/conf.d/puppetmaster.conf etc
against the documentation and I am not seeing any errors.

This seems like I have to be overlooking something really basic, and I'm
going to feel stupid when I find it, but it's right in my critical path
right now and I can't see it. Anyone have any suggestions? I can provide
config files and log files if need be, but I'm trying to avoid all the
redacting I'd need to do (my server is not literally named "caserver" etc).

Thanks

Pete

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Pete Hartman at Oct 2, 2013 at 2:34 am
    I have to do more testing to determine for certain, but it appears to
    have been some combination of
    1) the order in which modules were loaded, and
    2) not having mod_proxy_http loaded.




    On Tue, Oct 1, 2013 at 2:39 PM, Pete Hartman wrote:
    I am trying to establish what looks like a common pattern for scaling
    puppet. My main departure is that I'm using an F5 rather than an apache load
    balancer. Namely, I want to have my puppet agents go through the F5 to a
    pool of "master only" systems, and any Certificate activity to get proxied
    by those masters through to one single Certificate Authority. That CA
    system is not part of the F5 pool, it's role is to provide CA, Puppetdb and
    Postgresql. It is configured as a master because that was the easiest way
    to get a CA stood up, but I don't intend to use it as a master in normal
    operation (and in fact I don't plan to have it hosting any modules).

    I'm using RHEL 6, Apache, and Passenger, and Open Source Puppet.

    I initially set up passenger using puppetlabs/passenger from the Forge,
    (which got me most of the way there but not fully configured). All of these
    steps worked fine for the CA system to configure it as a working master (I
    have tested by registering systems with it, but then done puppet cert clean
    and wiped the test systems' ssl directories).

    I then set up my first master-only system the same way, except I didn't
    actually start the master service (as the docs say) until after I had set ca
    = false and ca_server = $MY_CA_SERVER into /etc/puppet.conf. I also made
    the necessary changes listed at
    http://docs.puppetlabs.com/guides/scaling_multiple_masters.html, including
    the certificate access on the CA system, the SSLProxyEngine on and
    ProxyPassMatch lines in the VHost definition in
    /etc/httpd/conf.d/puppetmaster.conf. I'm positive I followed all the steps
    in the docs in order, but I'm not having any luck with external agents.

    If I run puppet agent -t on the master-only system (with it's "server" in
    puppet.conf set to itself) it works fine--it can talk to the CA and talk to
    itself, and all is right with the world.

    If I run puppet agent -t on a client host, pointing at the load balancer's
    address (or even pointing direclty at the master-only system's real
    hostname), I get:

    [root@elmer ~]# puppet agent -t
    Info: Creating a new SSL key for elmer.allstate.com
    Error: Could not request certificate: Error 400 on SERVER: this master is
    not a CA
    Exiting; failed to retrieve certificate and waitforcert is disabled


    I've looked at the logs, enabled debug logging in the webserver with
    LogLevel, dug around everywhere I can think of, and I see no sign of any
    actual proxying going on. tcpdump certainly shows no attempt by the
    master-only system to contact the CA.

    What it LOOKS like is happening is that apache is not actually proxying
    anything, the request gets passed to the puppet master app running under
    passenger, and it (rightly) says "I'm not a CA" because
    /etc/puppet/puppet.conf says so.

    I do not see any errors in the logs about proxy attempts failing for this
    agent. I do see workers being attached for proxy purposes:

    [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1833): proxy: grabbed
    scoreboard slot 0 in child 27434 for worker
    https://caserver.allstate.com:8140/$1
    [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1852): proxy: worker
    https://caserver.allstate.com:8140/$1 already initialized
    [Tue Oct 01 13:48:26 2013] [debug] proxy_util.c(1949): proxy: initialized
    single connection worker 0 in child 27434 for caserver.allstate.com)


    I've repeatedly re-checked the settings in /etc/puppet.conf
    /etc/httpd/conf.d/passenger.conf, /et/chttpd/conf.d/puppetmaster.conf etc
    against the documentation and I am not seeing any errors.

    This seems like I have to be overlooking something really basic, and I'm
    going to feel stupid when I find it, but it's right in my critical path
    right now and I can't see it. Anyone have any suggestions? I can provide
    config files and log files if need be, but I'm trying to avoid all the
    redacting I'd need to do (my server is not literally named "caserver" etc).

    Thanks

    Pete

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Gavin Williams at Oct 2, 2013 at 7:08 am
    Pete

    I've not done this before, however am familiar with Puppet, and know a lot more about F5s...

    I note that you say that you're expecting apache on the masters to proxy onto the CA server.
    Is there any reason you couldn't use the F5 to select the CA server for any CA requests?
    Should be a fairly straight forward iRule to do pool selection based on the URI.

    Thoughts?

    Gav

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pete Hartman at Oct 2, 2013 at 12:36 pm
    I do not have responsibility for the F5's and I'm not sure what my
    networking team would be willing to do in terms of custom rules no
    matter how simple.

    The use of the apache proxy service on the masters is a configuration
    documented and recommended (at least as one alternative) by
    PuppetLabs; now that I have found what I was missing, I plan to stick
    with that.
    On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams wrote:
    Pete

    I've not done this before, however am familiar with Puppet, and know a lot more about F5s...

    I note that you say that you're expecting apache on the masters to proxy onto the CA server.
    Is there any reason you couldn't use the F5 to select the CA server for any CA requests?
    Should be a fairly straight forward iRule to do pool selection based on the URI.

    Thoughts?

    Gav

    --
    You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felipe Salum at Oct 2, 2013 at 7:27 pm
    Can you paste your /etc/httpd/conf.d/puppetmaster.conf ?
    On Wednesday, October 2, 2013 5:35:58 AM UTC-7, Pete Hartman wrote:

    I do not have responsibility for the F5's and I'm not sure what my
    networking team would be willing to do in terms of custom rules no
    matter how simple.

    The use of the apache proxy service on the masters is a configuration
    documented and recommended (at least as one alternative) by
    PuppetLabs; now that I have found what I was missing, I plan to stick
    with that.
    On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams wrote:
    Pete

    I've not done this before, however am familiar with Puppet, and know a
    lot more about F5s...
    I note that you say that you're expecting apache on the masters to proxy
    onto the CA server.
    Is there any reason you couldn't use the F5 to select the CA server for
    any CA requests?
    Should be a fairly straight forward iRule to do pool selection based on the URI.
    Thoughts?

    Gav

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    puppet-users...@googlegroups.com <javascript:>.
    To post to this group, send email to puppet...@googlegroups.com<javascript:>.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Pete Hartman at Oct 2, 2013 at 8:05 pm
    I tried to update this, but apparently failed.

    Problem was my own misunderstanding of apache.

    1) the passenger module was loaded before the proxy module, so the app
    was responding before apache could proxy the request
    2) I didn't recognize this as a working fix at first because I also
    omitted mod_proxy_http which was needed in addition to mod_proxy


    Thanks...

    Pete


    On Wed, Oct 2, 2013 at 2:27 PM, Felipe Salum wrote:
    Can you paste your /etc/httpd/conf.d/puppetmaster.conf ?

    On Wednesday, October 2, 2013 5:35:58 AM UTC-7, Pete Hartman wrote:

    I do not have responsibility for the F5's and I'm not sure what my
    networking team would be willing to do in terms of custom rules no
    matter how simple.

    The use of the apache proxy service on the masters is a configuration
    documented and recommended (at least as one alternative) by
    PuppetLabs; now that I have found what I was missing, I plan to stick
    with that.
    On Wed, Oct 2, 2013 at 2:08 AM, Gavin Williams wrote:
    Pete

    I've not done this before, however am familiar with Puppet, and know a
    lot more about F5s...

    I note that you say that you're expecting apache on the masters to proxy
    onto the CA server.
    Is there any reason you couldn't use the F5 to select the CA server for
    any CA requests?
    Should be a fairly straight forward iRule to do pool selection based on
    the URI.

    Thoughts?

    Gav

    --
    You received this message because you are subscribed to a topic in the
    Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    puppet-users...@googlegroups.com.
    To post to this group, send email to puppet...@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to a topic in the
    Google Groups "Puppet Users" group.
    To unsubscribe from this topic, visit
    https://groups.google.com/d/topic/puppet-users/xY5xnOU09Qg/unsubscribe.
    To unsubscribe from this group and all its topics, send an email to
    puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedOct 1, '13 at 7:39p
activeOct 2, '13 at 8:05p
posts6
users3
websitepuppetlabs.com

People

Translate

site design / logo © 2021 Grokbase