[Puppet Users] Analysing some puppetmaster logs to find out what's happening on an agent

On Tue, Apr 24, 2012 at 10:43:04AM +0800, Walter Heck wrote:
Hi all,

in an unfortunate incident, I managed to lock myself out of a client's
server. Basically an openssh module that by default disabled remote
root logins did that on a server that was only accessed by remote root
login (no other use raccounts present on that server). Unfortunately,
this is a colocated server and next trip to the dc is scheduled for
next thursday. No KVM over IP, no remote hands, pretty much the ideal
situation :P. The Xen server is still running, and so are the domU's
on it, but this is less then ideal. If any of the domU's goes down,
there's nothing we can do :)

Now, the puppet agent is running every 30 minutes, but something seems
to make it not execute the catalog. I have set the puppetmaster to
debug in order to see what's happening, but I can't figure it out.
Here's a gist of the puppet master log:
https://gist.github.com/2475554

x7 is the offending server, x6 has exactly the same puppet definition.
Can anyone tell me why the log for x7 just stops, with no error or
nothing? What does that indicate is happening on x7? Any help is much
appreciated :)

cheers,

--
Walter Heck

--
follow @walterheck on twitter to see what I'm up to!
--
Check out my new startup: Server Monitoring as a Service @ http://tribily.com
Follow @tribily on Twitter and/or 'Like' our Facebook page at
http://www.facebook.com/tribily

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

On Tue, Apr 24, 2012 at 10:43 AM, Walter Heck wrote:

Hi all,

in an unfortunate incident, I managed to lock myself out of a client's
server. Basically an openssh module that by default disabled remote
root logins did that on a server that was only accessed by remote root
login (no other use raccounts present on that server). Unfortunately,
this is a colocated server and next trip to the dc is scheduled for
next thursday. No KVM over IP, no remote hands, pretty much the ideal
situation :P. The Xen server is still running, and so are the domU's
on it, but this is less then ideal. If any of the domU's goes down,
there's nothing we can do :)

Now, the puppet agent is running every 30 minutes, but something seems
to make it not execute the catalog. I have set the puppetmaster to
debug in order to see what's happening, but I can't figure it out.
Here's a gist of the puppet master log:
https://gist.github.com/2475554

x7 is the offending server, x6 has exactly the same puppet definition.
Can anyone tell me why the log for x7 just stops, with no error or
nothing? What does that indicate is happening on x7? Any help is much
appreciated :)

cheers,

--
Walter Heck

--
follow @walterheck on twitter to see what I'm up to!
--
Check out my new startup: Server Monitoring as a Service @
http://tribily.com
Follow @tribily on Twitter and/or 'Like' our Facebook page at
http://www.facebook.com/tribily

--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Apr 24, 2012 at 1:12 PM, Walter Heck wrote:

Hi Sharuzzaman,

On Tue, Apr 24, 2012 at 10:59, Sharuzzaman Ahmat Raslan
wrote:
I checked and it seems the last report in there is from July 18th. It
contains a bunch of errors from when it had the full puppet manifests
enabled, but that should be irrelevant now as we only have a single
exec in there. Any idea as to why it's not creating new reports?

cheers,

--
Walter Heck

--
follow @walterheck on twitter to see what I'm up to!
--
Check out my new startup: Server Monitoring as a Service @
http://tribily.com
Follow @tribily on Twitter and/or 'Like' our Facebook page at
http://www.facebook.com/tribily

--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Apr 23, 9:43 pm, Walter Heck wrote:
Hi all,

in an unfortunate incident, I managed to lock myself out of a client's
server. Basically an openssh module that by default disabled remote
root logins did that on a server that was only accessed by remote root
login (no other use raccounts present on that server). Unfortunately,
this is a colocated server and next trip to the dc is scheduled for
next thursday. No KVM over IP, no remote hands, pretty much the ideal
situation :P. The Xen server is still running, and so are the domU's
on it, but this is less then ideal. If any of the domU's goes down,
there's nothing we can do :)

Now, the puppet agent is running every 30 minutes, but something seems
to make it not execute the catalog. I have set the puppetmaster to
debug in order to see what's happening, but I can't figure it out.
Here's a gist of the puppet master log:https://gist.github.com/2475554

x7 is the offending server, x6 has exactly the same puppet definition.
Can anyone tell me why the log for x7 just stops, with no error or
nothing? What does that indicate is happening on x7? Any help is much
appreciated :)

On Tue, Apr 24, 2012 at 9:09 AM, jcbollinger wrote:



In the excerpt you posted, it looks like x7 is getting a cached
catalog, whereas x6's catalog needed to be recompiled. The fact that
their manifests are the same is not inconsistent with that. Perhaps
that's why you don't see more.

Alternatively, most of the log lines pertaining to x7 appear to show
it downloading plugins -- maybe you have a hung client, but successive
cron-initiated runs are performing plugin sync and fact gathering
before that stops them.

Since it looks like the client is still (plugin)syncing, however, that
may be enough of an opening for you to break the server back open.
You could try sending it a custom fact that has whatever clever side
effect you like. I'm not certain whether facts are evaluated with
privilege, but you should at least be able to collect information and
write it to your share.


John

--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Apr 25, 2012 at 01:18, Michael Baydoun wrote:
Maybe you could use puppet to configure send to a remote syslog.  If that
works, you could then add your problem module back, and get visibility to
exactly what the client is doing, and any errors it is seeing.

Or make some small modification to the existing log file on the remote
server, with filebucketing to your master, and use that to retrieve the
puppet log from the remote

--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 28 April 2012 15:11, Walter Heck wrote:

So, just to make sure I understand this correctly: in this case just
removing all the puppet code didn't help, since the mere existence of
the module with the custom fact in our module path made the fact
execute on the agent, right?