FAQ
As mentioned in my previous thread at
http://groups.google.com/group/puppet-users/browse_thread/thread/d491889c89c1a0e0/2ba280696d1ecf96?lnk=gst&q=matthew+burgess#2ba280696d1ecf96,
I'm seeing intermittent errors on my nodes of the form:

Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>' for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>' for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>' for
["2.7.14", "2.7.14"]:Array on node master.domain.com
Wed May 30 18:02:24 +0100 2012 Puppet (err): undefined method `>' for
["2.7.14", "2.7.14"]:Array on node master.domain.com

Upon further investigation I've managed to, I think, finger the blame
on storeconfigs.

I've set up a new puppet master, completely segregated from the rest
of the puppet infrastructure, so there is a total of
1 puppet master and 1 puppet client in this environment. An ENC is
configured that currently states that this node should
not have any manifests applied.

With storeconfigs set to true, and pointing to a MySQL database on the
same host, I can reliably trigger the issue using
puppet-load.rb.

As soon as I set storeconfigs to false, and restart httpd (puppet is
being served behind mod_passenger), a puppet-load
run with 100 concurrent clients works with 100% success.

Now, the storeconfigs instructions at
http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration
mentions the use of a message broker, which was going to be by next
plan of attack, but I guess at the moment I'm just kind of interested
to see if anyone else has hit problems similar to mine and what kind
of scaling they've achieved using plain storeconfigs without a message
broker.

Thanks,

Matt.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Matthew Burgess at Jun 13, 2012 at 2:28 pm

    On Wed, Jun 13, 2012 at 2:53 PM, Matthew Burgess wrote:

    As soon as I set storeconfigs to false, and restart httpd (puppet is
    being served behind mod_passenger), a puppet-load
    run with 100 concurrent clients works with 100% success.
    Sorry, this should read 20 concurrent clients. 100 concurrent clients
    happens to work as well, but
    that's way above the level of scaling we require at present.

    In addition, the tests were run after performing a 'puppetd --test' to
    store the initial config
    in the DB. With no manifests being applied, I'm not expecting any
    further writes to the DB during
    the test runs. Is that a reasonable assumption, and therefore is it
    also reasonable to assume
    that I should be focusing my efforts on mysql read optimisations/tuning?

    Regards,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jared Ballou at Jun 13, 2012 at 2:58 pm
    I ran into this issue when Active(Record|Model|Support) had a version
    higher than 3.0.11. I'm running RHEL6 x64 with Ruby 1.8.7 from RPM and
    MySQL gem 2.8.1 built from source, if that rings true for you. Which
    versions of the installed gems do you have? And if you run the puppet
    master as
    puppet master --no-daemonize -vd
    does it give any pertinent output for us to look at?

    As for data reads/writes, every time a node checks in, if storedconfigs
    is on, it will update the facts and some other ancillary information in
    the database. I've got about 400 nodes in my setup and it's still not
    very demanding on MySQL even with quite a lot of data going in and out,
    so the stock config values should suffice while you spend time tracking
    down all the other issues with getting the environment stood up.

    Good luck!

    -Jared
    On 06/13/2012 10:28 AM, Matthew Burgess wrote:
    On Wed, Jun 13, 2012 at 2:53 PM, Matthew Burgess
    wrote:
    As soon as I set storeconfigs to false, and restart httpd (puppet is
    being served behind mod_passenger), a puppet-load
    run with 100 concurrent clients works with 100% success.
    Sorry, this should read 20 concurrent clients. 100 concurrent clients
    happens to work as well, but
    that's way above the level of scaling we require at present.

    In addition, the tests were run after performing a 'puppetd --test' to
    store the initial config
    in the DB. With no manifests being applied, I'm not expecting any
    further writes to the DB during
    the test runs. Is that a reasonable assumption, and therefore is it
    also reasonable to assume
    that I should be focusing my efforts on mysql read optimisations/tuning?

    Regards,

    Matt.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matthew Burgess at Jun 13, 2012 at 3:50 pm

    On Wed, Jun 13, 2012 at 3:58 PM, Jared Ballou wrote:
    I ran into this issue when Active(Record|Model|Support) had a version
    higher than 3.0.11. I'm running RHEL6 x64 with Ruby 1.8.7 from RPM and
    MySQL gem 2.8.1 built from source, if that rings true for you. Which
    versions of the installed gems do you have? And if you run the puppet
    master as
    puppet master --no-daemonize -vd
    does it give any pertinent output for us to look at?
    Thanks for the quick response, Jared. It all sounds far too familiar :-)

    I'm running a RHEL 5.8 x86_64 server here. Ruby is 1.8.7-299 from
    aegisco's repository and rubygems-1.8.5 from the same location.

    I did try with activerecord-3.2.6 but quickly ran into
    http://projects.puppetlabs.com/issues/9290 so downgraded to 3.0.15.
    Based on your message I've tried both 3.0.11 and 3.0.10 and still have
    the same results.

    All gems are installed via 'gem install --local --verbose <gem name>'

    $ gem list

    *** LOCAL GEMS ***

    activemodel (3.0.10)
    activerecord (3.0.10)
    activesupport (3.0.10)
    addressable (2.2.8)
    arel (2.0.10)
    builder (2.1.2)
    cookiejar (0.3.0)
    daemon_controller (1.0.0)
    em-http-request (1.0.2)
    em-socksify (0.2.0)
    eventmachine (1.0.0.beta.4)
    fastthread (1.0.7)
    http_parser.rb (0.5.3)
    i18n (0.5.0)
    json (1.7.3)
    multi_json (1.3.6)
    mysql (2.8.1)
    passenger (3.0.12)
    rack (1.4.1)
    rake (0.9.2.2)
    tzinfo (0.3.,33)

    In terms of output from the puppet master, I get nothing of note with
    those options. If I try with:

    # puppet master --no-daemonize --debug --trace --verbose --logdest
    /tmp/puppetmaster.log

    I get the results in the attached log.
    As for data reads/writes, every time a node checks in, if storedconfigs
    is on, it will update the facts and some other ancillary information in
    the database. I've got about 400 nodes in my setup and it's still not
    very demanding on MySQL even with quite a lot of data going in and out,
    so the stock config values should suffice while you spend time tracking
    down all the other issues with getting the environment stood up.
    That's all useful info, thanks!

    Regards,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Kinzel, David at Jun 13, 2012 at 3:54 pm

    On Wed, Jun 13, 2012 at 3:58 PM, Jared Ballou
    wrote:
    I ran into this issue when Active(Record|Model|Support) had a version
    higher than 3.0.11. I'm running RHEL6 x64 with Ruby 1.8.7
    from RPM and
    MySQL gem 2.8.1 built from source, if that rings true for you. Which
    versions of the installed gems do you have? And if you run the puppet
    master as
    puppet master --no-daemonize -vd
    does it give any pertinent output for us to look at?
    Thanks for the quick response, Jared. It all sounds far too
    familiar :-)

    I'm running a RHEL 5.8 x86_64 server here. Ruby is 1.8.7-299 from
    aegisco's repository and rubygems-1.8.5 from the same location.

    I did try with activerecord-3.2.6 but quickly ran into
    http://projects.puppetlabs.com/issues/9290 so downgraded to 3.0.15.
    Based on your message I've tried both 3.0.11 and 3.0.10 and still have
    the same results.
    Are you perhaps running into a max db connection issue (this should be recorded in logs).

    We set something like wait_timeout=60 in our mysql my.cnf to kill "idle" connections that don't seem to get properly reused.


    This email communication and any files transmitted with it may contain
    confidential and or proprietary information and is provided for the use of the
    intended recipient only. Any review, retransmission or dissemination of this
    information by anyone other than the intended recipient is prohibited. If you
    receive this email in error, please contact the sender and delete this
    communication and any copies immediately. Thank you.

    http://www.encana.com

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matthew Burgess at Jun 13, 2012 at 4:09 pm

    On Wed, Jun 13, 2012 at 4:54 PM, Kinzel, David wrote:

    Are you perhaps running into a max db connection issue (this should be recorded in logs).

    We set something like wait_timeout=60 in our mysql my.cnf to kill "idle" connections that don't seem to get properly reused.
    I don't think so. I can reproduce the failure by restarting the
    puppet master, then running puppet-load.rb with '--concurrency 2'
    *twice*. The first time will always succeed; both clients complete
    successfully. The 2nd and subsequent invocations will always fail
    with the 400 error.

    If anyone here is able to share their production apache vhost configs
    including any passenger directives, I'd be grateful, just so I can
    compare with mine that were cobbled together via various online
    sources.

    Regards,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matthew Burgess at Jun 14, 2012 at 11:29 am
    So, continuing this saga, I decided yesterday to try and make use of
    puppetdb as it's supposed to have performance improvements over and
    above storeconfigs.

    My first change was to move from a MySQL to PostgreSQL database behind
    storeconfigs. That still showed the issues I'm seeing. With
    PostgreSQL in place, I then configured puppetdb, but I'm still seeing
    these issues with concurrent users.

    At this point, is it worth raising an issue at puppetlabs.com? I have
    the failing environment on a VM now, so if someone wants a copy/or
    login details to it, I'm happy to provide them if it helps figure out
    what's going on.

    Thanks,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Chris Price at Jun 14, 2012 at 12:38 pm
    Hi Matt,

    I think it would be worth opening a ticket for this (
    http://projects.puppetlabs.com/projects/puppet/issues/new ). The fact that
    you are seeing it with both storeconfigs/mysql and puppetdb/postgres leads
    me to believe that the issue is more likely in puppet core than in either
    of the catalog storage backends.

    It would also be useful if you could run the puppet master with the
    "--trace" command-line option (and possibly with --debug as well) and
    capture that log file.

    It sounds like you are running the master via apache/passenger, correct?

    A manifest file that could reproduce your setup would be ideal to attach to
    the ticket, but the VM will be very useful if it's not easy to put together
    such a manifest. How large is the VM image? It might be too big to attach
    directly to the ticket, but if you mention it in the ticket we can figure
    out the best way to get the file from you.

    Thanks!
    Chris
    On Thursday, June 14, 2012 4:29:17 AM UTC-7, Matthew Burgess wrote:

    So, continuing this saga, I decided yesterday to try and make use of
    puppetdb as it's supposed to have performance improvements over and
    above storeconfigs.

    My first change was to move from a MySQL to PostgreSQL database behind
    storeconfigs. That still showed the issues I'm seeing. With
    PostgreSQL in place, I then configured puppetdb, but I'm still seeing
    these issues with concurrent users.

    At this point, is it worth raising an issue at puppetlabs.com? I have
    the failing environment on a VM now, so if someone wants a copy/or
    login details to it, I'm happy to provide them if it helps figure out
    what's going on.

    Thanks,

    Matt.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/NdraCH3guVAJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matthew Burgess at Jun 14, 2012 at 1:04 pm

    On Thu, Jun 14, 2012 at 1:31 PM, Chris Price wrote:
    Hi Matt,

    I think it would be worth opening a ticket for this
    ( http://projects.puppetlabs.com/projects/puppet/issues/new ).  The fact
    that you are seeing it with both storeconfigs/mysql and puppetdb/postgres
    leads me to believe that the issue is more likely in puppet core than in
    either of the catalog storage backends.
    Thanks Chris. I'll open a ticket when I can step away from the
    firefighting for a moment :-)

    In the meantime, just to clarify, I'm seeing this in
    storeconfigs/mysql, storeconfigs/postgres
    and puppetdb/postgres. As previously mentioned though, I only see
    this with storeconfigs=true
    and never with storeconfigs=false. So maybe it is in puppet core
    somewhere, but only in an
    area that storeconfigs touches?
    It would also be useful if you could run the puppet master with the
    "--trace" command-line option (and possibly with --debug as well) and
    capture that log file.
    OK, will add that log to the ticket.
    It sounds like you are running the master via apache/passenger, correct?
    Yes, but I think it's reproducible outside of apache/passenger too,
    though it's a bit hard to
    tickle there due to Webrick crapping out with a different issue when
    relatively lightly loaded.
    A manifest file that could reproduce your setup would be ideal to attach to
    the ticket, but the VM will be very useful if it's not easy to put together
    such a manifest.
    That's the thing, this is being triggered on a puppetmaster with *no*
    manifests; determined
    after originally thinking that one of my manifests was to blame, I got
    rid of the lot and could
    still trigger it!
    How large is the VM image?  It might be too big to attach
    directly to the ticket, but if you mention it in the ticket we can figure
    out the best way to get the file from you.
    It'll be in the order of 1GB I think. As there's no sensitive details
    in that environment though
    I think I'll attach puppet.conf, auth.conf and the httpd configs too
    so that someone can try
    to reproduce. Hopefully that's enough, otherwise I'll have another
    look at getting the
    VM uploaded or otherwise made available somehow.

    Regards,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matthew Burgess at Jun 18, 2012 at 4:50 pm

    On Thu, Jun 14, 2012 at 1:31 PM, Chris Price wrote:
    Hi Matt,

    I think it would be worth opening a ticket for this
    ( http://projects.puppetlabs.com/projects/puppet/issues/new ).
    Hi Chris,

    This is now #15086. Hopefully there's enough info on the ticket to reproduce.

    The VM is 1.5GB in size, so may be a little awkward to make available
    to anyone working on the ticket, but I'm happy to give anything a try!

    Regards,

    Matt.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedJun 13, '12 at 1:53p
activeJun 18, '12 at 4:50p
posts10
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase