FAQ
After restarting our puppetmaster host, we are seeing high CPU load and a
ton of queued passenger requests. I would suspect that our puppetmaster is
falling victim to the thundering herd of puppet agents, but we have
deployed most of our agents using a randomized cron job. The others are
using 'splay = true' if they are running puppet agent.

Some info:
Puppetmaster is RHEL6 x86_64 (VM)
4 vCPU 8GB memory
Apache 2.2
Passenger 3.0.7

The vhost for our puppetmaster includes these passenger options:
PassengerHighPerformance on
PassengerUseGlobalQueue on
PassengerPoolIdleTime 1500
PassengerMaxPoolSize 24
PassengerMaxRequests 4000
PassengerStatThrottleRate 120
PassengerSpawnMethod smart
RackAutoDetect Off
RailsAutoDetect Off


Look at all of these queued requests!

Passenger-status:
----------- General information -----------
max = 24
count = 24
active = 22
inactive = 2
Waiting on global queue: 234

----------- Application groups -----------
/etc/puppet/rack:
App root: /etc/puppet/rack
* PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s
* PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s
* PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s
* PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s
* PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s
* PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s
* PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s
* PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s
* PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s
* PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s
* PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s
* PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s
* PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s
* PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s
* PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s
* PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s
* PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s
* PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s
* PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s
* PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s
* PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s
* PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s

/usr/share/foreman:
App root: /usr/share/foreman
* PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s
* PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/p1uXXzKJhCcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Luke Baker at Jul 31, 2012 at 9:17 pm
    I ended up restart the mysql database on the storeconfigs server which
    helped this problem dramatically. But, I'm still open for more tuning
    options if they're out there : )
    On Tuesday, July 31, 2012 8:34:45 AM UTC-5, Luke Baker wrote:

    After restarting our puppetmaster host, we are seeing high CPU load and a
    ton of queued passenger requests. I would suspect that our puppetmaster is
    falling victim to the thundering herd of puppet agents, but we have
    deployed most of our agents using a randomized cron job. The others are
    using 'splay = true' if they are running puppet agent.

    Some info:
    Puppetmaster is RHEL6 x86_64 (VM)
    4 vCPU 8GB memory
    Apache 2.2
    Passenger 3.0.7

    The vhost for our puppetmaster includes these passenger options:
    PassengerHighPerformance on
    PassengerUseGlobalQueue on
    PassengerPoolIdleTime 1500
    PassengerMaxPoolSize 24
    PassengerMaxRequests 4000
    PassengerStatThrottleRate 120
    PassengerSpawnMethod smart
    RackAutoDetect Off
    RailsAutoDetect Off


    Look at all of these queued requests!

    Passenger-status:
    ----------- General information -----------
    max = 24
    count = 24
    active = 22
    inactive = 2
    Waiting on global queue: 234

    ----------- Application groups -----------
    /etc/puppet/rack:
    App root: /etc/puppet/rack
    * PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s
    * PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s
    * PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s
    * PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s
    * PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s
    * PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s
    * PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s
    * PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s
    * PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s
    * PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s
    * PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s
    * PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s
    * PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s
    * PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s
    * PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s
    * PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s
    * PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s
    * PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s
    * PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s
    * PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s
    * PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s
    * PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s

    /usr/share/foreman:
    App root: /usr/share/foreman
    * PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s
    * PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/hNtnLgeBLzYJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ramin K at Aug 1, 2012 at 12:45 am

    On 7/31/2012 2:17 PM, Luke Baker wrote:
    I ended up restart the mysql database on the storeconfigs server which
    helped this problem dramatically. But, I'm still open for more tuning
    options if they're out there : )

    Some info:
    Puppetmaster is RHEL6 x86_64 (VM)
    4 vCPU 8GB memory
    Apache 2.2
    Passenger 3.0.7
    PassengerHighPerformance on
    PassengerMaxPoolSize 8
    PassengerMinInstances 1
    PassengerMaxRequests 10000
    PassengerStatThrottleRate 120
    PassengerPreStart https://puppet.mydomain.com:8140/

    In Passenger 3.0.x GlobalQueue is on by default so you can drop that.

    MinInstances is new in 3.0.x. Using it obviates the need for timeout
    settings. Setting MinInstances beyond 1 unless done at the vhost level
    for specific reasons is a bad idea because Passenger can balance
    application instances better than you. Keep in mind that MinInstances
    does not start applications when your http daemon starts for the first
    time.

    PassengerPreStart does start applications when the server restarts and
    takes a URL as an argument which it uses in the host headers when it
    connects to localhost. This should also be applied at the vhost level.

    MaxRequests can be reasonably high with a mature app like Puppet.

    MaxPoolSize is a little harder to size. A Puppet master is usually CPU
    bound and I default to (cores * 2), but it depends on your apps. I'd aim
    for 80% of CPU with all Rack processes in use and add or drop from the
    poolsize to achieve it. Keep in mind that a Puppet Rack process will
    weigh in around 150-200MB and you should have enough RAM to support the
    max pool size.

    Rails/Rack autodetect shouldn't be needed with a normal vhost for Puppet.

    I would not touch the Passenger spawn method at all unless there is a
    specific reason to modify it. There is not a reason to modify it with
    Puppet in my opinion.
    http://www.modrails.com/documentation/Users%20guide%20Apache.html#spawning_methods_explained

    Ramin

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jake - USPS at Aug 2, 2012 at 1:35 pm
    If that seemed to help maybe you need to tune your mysql server(s)? I've
    tuned mine quite a bit from default OS distro settings using
    http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert.

    I've also separated mysql to a different server then what my puppetmasters
    run on.

    Regards,
    Jake

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zo7mWV3ehGAJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Ad at Aug 2, 2012 at 7:17 pm

    On Thursday, August 2, 2012 8:34:55 AM UTC-5, Jake - USPS wrote:
    If that seemed to help maybe you need to tune your mysql server(s)? I've
    tuned mine quite a bit from default OS distro settings using
    http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert.

    Check out Sheeri K. Cabral's fork here:

    https://launchpad.net/mysqltuner

    Regards,

    Adam

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/SVmfRSktCOwJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedJul 31, '12 at 1:34p
activeAug 2, '12 at 7:17p
posts5
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase