[Puppet Users] RHEL patches broke my puppetd

You should upgrade from 0.25.5. It is quite old and no longer supported.

How about stopping the agent before performing OS upgrades (as opposed to
just applying updates released for the current OS version)?


Just to be sure: you're running that with privilege, right?


If by "troubleshoot this" you mean get Puppet working correctly again, then
there is probably no alternative to forcing Puppet stopped and then
removing the lock file. That might involve manually killing a stalled
puppetd. If you prefer, that could take the form of restarting the whole
server, after which it would be safe to remove the lock file without
shutting down the agent. You really should not remove the lock file,
neither manually nor via "puppetd --enable", while the puppetd process that
created it is alive, however.

If by "troubleshoot this" you mean determine what went wrong and why, then
you need to gather information, including:

    - The actual state of the system. Is the agent in fact running? Is the
    lock file in fact present?
    - The logs of the most recent Puppet activity not related to your failed
    / skipped runs and diagnostic efforts. What did puppet last do -- or what
    was it in the process of doing -- when it entered the state it is now in?
    - The updates that were actually applied to get from your (possibly
    updated) RHEL 5.5 to the current (possibly updated) 5.6.

Then you need to conduct an analysis, on which I cannot advise you in any
detail. I think it more likely that the update clobbered application of
some resource, causing the agent to stall in a manner tied to the resource
type and its chosen provider, than that the update clobbered the core
puppet engine. But I can't be sure.

Ultimately, even if you are able to form a good hypothesis about what
happened, and even if you are able to test that hypothesis to prove its
plausibility, I don't know any way to be *certain* that what you come up
with is the correct explanation for what actually happened. You'll have to
use your best judgement.


John

This did not occur to me. Sounds Windows-y... LOL

Yes, running as root.

Thanks for going into detail about this. I too believe that some puppet
resource was indeed stepped on by the RHEL update from 5.5 to 5.6. I am in
the process of reverting the ruby packages on the affected servers from
1.8.5.-19 to 1.8.5.-5. Also related, I discovered that the puppet listener
wasn't properly restarting itself due to the control script not specifying
the correct location of the PID file. That issue has been fixed so I no
longer have to manually kill the process. Will update the thread later
this week with progress

Thanks
DS

A bit, maybe, but if it were really Windows-y then you'd need to reboot the
system after the upgrade. :-)

Conceptually, it doesn't seem unreasonable to me to avoid attempting to
perform conflicting actions concurrently. An OS upgrade certainly is at
least a potential conflict with Puppet-controlled configuration
management. It might be possible to harmonize these (for example, by
making Puppet perform the upgrade), but it's hard to be sure without
knowing the nature of the problem that occurred.

[...]

I would be very surprised if ruby-1.8.5-19 were the problem. Indeed, I
have multiple CentOS 5.x systems under Puppet management, and I have never
had a problem with any of the official CentOS Ruby packages. If you
contemplate moving up to Puppet 2.x or higher in the future, however, then
at that time you will need at least Ruby 1.8.7. That would need to be
provided by a third-party package, given CentOS / RHEL policy on software
versions.

Indeed do please update us about what you find. I'm curious.


John

Finally made progress today, the answer for me was to back out of the newer
kernel version as mentioned here:
https://groups.google.com/d/msg/puppet-users/ho5Thsm5q1E/FPe4N9KvhD4J

After downgrading from 2.6.18-238.49.1.el5 back to 2.6.18-194.32.1.el5 my
puppetd is functioning properly again. Something about puppet kick was
causing the puppet client to get stuck indefinitely until manually killed
or restarted via init script.

So, locally puppetd worked fine *until* a puppetrun from the master
happened, causing the client to enter a stuck state.

Now I just have to upgrade the puppetmaster in a test environment and hope
my code still works!! I am probably going to try baby steps and go to the
latest 2.6 instead of straight to 3.x

Thanks for the help!
Dave