FAQ
I've been asked to look at a problem on an overseas rig whereby certain
bits of config were going awry. This was down to the fact that they were
querying the mcollective registration database and feeding back in to the
configs, using queries based on class data. The mcollective + mongodb setup
is working fine, but it would appear that the classes.txt file which
mcollective reads to get configuration management classes is, on certain
servers, spuriously being emptied of all classes other than 'settings'

Seeing as the settings class is being left in, I can only assume that it
must be puppet doing the modification. Has anyone ever seen behaviour like
this before? It's truly bizarre, and even stranger is that after this has
happened, sometimes the classes populate again. Every time I've run puppet
manually with puppetd --test or run it --noop, it's run just fine and
without noop it always seems to fix the problem, but later the node will
show up as having only one class again and even later fix itself, so it's
next to impossible to pin down.

This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zi9NP9ysFkcJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Search Discussions

  • Josh Cooper at Oct 19, 2012 at 5:09 pm
    Hi Matt,
    On Fri, Oct 19, 2012 at 8:55 AM, Matt Carroll wrote:
    I've been asked to look at a problem on an overseas rig whereby certain bits
    of config were going awry. This was down to the fact that they were querying
    the mcollective registration database and feeding back in to the configs,
    using queries based on class data. The mcollective + mongodb setup is
    working fine, but it would appear that the classes.txt file which
    mcollective reads to get configuration management classes is, on certain
    servers, spuriously being emptied of all classes other than 'settings'

    Seeing as the settings class is being left in, I can only assume that it
    must be puppet doing the modification. Has anyone ever seen behaviour like
    this before? It's truly bizarre, and even stranger is that after this has
    happened, sometimes the classes populate again. Every time I've run puppet
    manually with puppetd --test or run it --noop, it's run just fine and
    without noop it always seems to fix the problem, but later the node will
    show up as having only one class again and even later fix itself, so it's
    next to impossible to pin down.

    This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To view this discussion on the web visit
    https://groups.google.com/d/msg/puppet-users/-/zi9NP9ysFkcJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to
    puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at
    http://groups.google.com/group/puppet-users?hl=en.
    There was this issue (http://projects.puppetlabs.com/issues/4518),
    which was fixed in early 2.6. However, I think RI has said he's seen
    this more recently. You may want to ask on the mcollective-users list
    if you don't get a reply here.

    Josh

    --
    Josh Cooper
    Developer, Puppet Labs

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matt Carroll at Oct 22, 2012 at 9:43 am
    Hmm, I'm not sure that this is the same issue... the issue details the classes.txt only containing a newline wheras I'm experiencing a problem whereby the file sometimes only contains the default class 'settings'. It appears as if the node is compiling spuriously against a different environment with an empty default class every now and again.

    I'm going to look in to the possibility Of upgrading the rig, as it may fix the problem, and seeing as it's quite an old version it's not really too necessary to know exactly what was causing the behaviour.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/3NhJKOAfRjQJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • R.I.Pienaar at Oct 22, 2012 at 9:46 am

    ----- Original Message -----
    From: "Matt Carroll" <oholiab@gmail.com>
    To: puppet-users@googlegroups.com
    Sent: Monday, October 22, 2012 10:43:44 AM
    Subject: Re: [Puppet Users] Puppet spuriously removing classes from classes.txt

    Hmm, I'm not sure that this is the same issue... the issue details
    the classes.txt only containing a newline wheras I'm experiencing a
    problem whereby the file sometimes only contains the default class
    'settings'. It appears as if the node is compiling spuriously
    against a different environment with an empty default class every
    now and again.

    I'm going to look in to the possibility Of upgrading the rig, as it
    may fix the problem, and seeing as it's quite an old version it's
    not really too necessary to know exactly what was causing the
    behaviour.
    depending on version it also writes resources.txt, so you can determine
    if when classes.txt is empty if it really was not managing any resources.

    Being someone who cares deeply for classes.txt, I've not observed this
    behavior

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Jcbollinger at Oct 22, 2012 at 1:48 pm

    On Friday, October 19, 2012 10:55:03 AM UTC-5, Matt Carroll wrote:
    I've been asked to look at a problem on an overseas rig whereby certain
    bits of config were going awry. This was down to the fact that they were
    querying the mcollective registration database and feeding back in to the
    configs, using queries based on class data. The mcollective + mongodb setup
    is working fine, but it would appear that the classes.txt file which
    mcollective reads to get configuration management classes is, on certain
    servers, spuriously being emptied of all classes other than 'settings'

    Seeing as the settings class is being left in, I can only assume that it
    must be puppet doing the modification. Has anyone ever seen behaviour like
    this before? It's truly bizarre, and even stranger is that after this has
    happened, sometimes the classes populate again. Every time I've run puppet
    manually with puppetd --test or run it --noop, it's run just fine and
    without noop it always seems to fix the problem, but later the node will
    show up as having only one class again and even later fix itself, so it's
    next to impossible to pin down.

    This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

    Are you certain the behavior is buggy? The fact that Puppet rights itself
    makes me think that Puppet itself is not the problem.

    What if there is a cron job (or a human admin) that periodically does
    something like "puppetd --onetime --no-daemonize --tags settings"? Or what
    if the affected node's manifests use schedules in a way that occasionally
    excludes all classes except 'settings'? The former, at least, should show
    up in the system log.

    You might also get some insight by wrapping puppetd in a script that makes
    a timestamped backup of classes.txt after each run. Done right, that ought
    to establish whether Puppet is the one writing the unexpected content, and
    if so then the timestamps when the bad files appear might tell you
    something useful. Perhaps you could dredge the same information out of
    your system logs, but the timestamped backups might be easier to analyze
    (and they would pinpoint where in your logs to look for additional
    information).


    John

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/1RlbAp6sTdMJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
  • Matt Carroll at Oct 25, 2012 at 9:23 am
    I'm not sure whether it's buggy behaviour or not, but by comparing the
    daemon.log and an inotify left running on the file I can confirm that it's
    puppet that's making the changes. Whether or not it's someone running
    puppet against a stupid environment or not remains to be seen, but I still
    need to find out what is going on...

    Like I say, this is not directly my infrastructure or I'd have a better
    idea.
    On Monday, 22 October 2012 14:41:16 UTC+1, jcbollinger wrote:


    On Friday, October 19, 2012 10:55:03 AM UTC-5, Matt Carroll wrote:

    I've been asked to look at a problem on an overseas rig whereby certain
    bits of config were going awry. This was down to the fact that they were
    querying the mcollective registration database and feeding back in to the
    configs, using queries based on class data. The mcollective + mongodb setup
    is working fine, but it would appear that the classes.txt file which
    mcollective reads to get configuration management classes is, on certain
    servers, spuriously being emptied of all classes other than 'settings'

    Seeing as the settings class is being left in, I can only assume that it
    must be puppet doing the modification. Has anyone ever seen behaviour like
    this before? It's truly bizarre, and even stranger is that after this has
    happened, sometimes the classes populate again. Every time I've run puppet
    manually with puppetd --test or run it --noop, it's run just fine and
    without noop it always seems to fix the problem, but later the node will
    show up as having only one class again and even later fix itself, so it's
    next to impossible to pin down.

    This is 2.6.7 on Ubuntu running against a centralised puppetmaster.

    Are you certain the behavior is buggy? The fact that Puppet rights itself
    makes me think that Puppet itself is not the problem.

    What if there is a cron job (or a human admin) that periodically does
    something like "puppetd --onetime --no-daemonize --tags settings"? Or what
    if the affected node's manifests use schedules in a way that occasionally
    excludes all classes except 'settings'? The former, at least, should show
    up in the system log.

    You might also get some insight by wrapping puppetd in a script that makes
    a timestamped backup of classes.txt after each run. Done right, that ought
    to establish whether Puppet is the one writing the unexpected content, and
    if so then the timestamps when the bad files appear might tell you
    something useful. Perhaps you could dredge the same information out of
    your system logs, but the timestamped backups might be easier to analyze
    (and they would pinpoint where in your logs to look for additional
    information).


    John
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/GWaww-ftBBsJ.
    To post to this group, send email to puppet-users@googlegroups.com.
    To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
    For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedOct 19, '12 at 4:28p
activeOct 25, '12 at 9:23a
posts6
users4
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase