On Thursday, August 8, 2013 12:48:03 PM UTC+1, Ken Barber wrote:

No good idea yet, but there is something suspicious in your curl
responses - the "resource" hash, did you obfuscate this yourself on
purpose? The two hashes between the first and second requests are
identical. That hash is calculated based on the sum of the resource,
including parameters - so it seems impossible that PuppetDB arrived at
the same hash with and without parameters.
I think that's just me being too sensorship heavy and abusing copy and
paste, I would have copied some fields from the same example. Trust me that
the resources dictionary was empty though ;-)

Maybe just maybe the responses were identical, and somehow PuppetDB
was not returning parameters as a fault. This might indicate some sort
of integrity problem, whereby the link to the parameters in the RDBMS
was lost somehow, although this is the first time I've heard of it
Luke. Maybe this was an upgrade schema change failure between 1.0.1
and 1.3.2?

We thought of that, however we upgraded 3 weeks ago and I saw working
reports in my Dashboard on the Nagios server up until Friday last week, so
it was able to collect valid resources for a long while after the DB
migration. What happened on Friday though was I basically had this running:
the database
mco rpc shellout cmd cmd="puppet agent --test --noop" -T mcollective

Then I realised we had broken manifests everywhere, so I fixed it and
started MC again. From that point onward the Nagios server was unable to
collect as it was getting these dodgy resources from PuppetDB. What doesn't
make sense is my subsequent site wide no-op should have replaced every
manifest in PuppetDB, so I'm stumped.

I'd have to consult what changed in the schema between
those two points to determine if this is likely however. Does the
timing of your upgrade, and the first time you saw this fault line up
with such a possibility? Remember a schema change will only occur
after a restart of PuppetDB ... (so perhaps consult your logs to see
when this happened after upgrade).
Just checked - PuppetDB restarted at the same time the RPMs were upgraded
and hasn't restarted since.

Let me at least try to replicate while I await your responses.

Now if I was thinking smart I would have taken a Postgres backup before I
re-freshed all the catalogs, but I didn't, not sure if that would have
helped much. I agree with subsequent posts as well - probably not a
migration problem.

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 9 | next ›
Discussion Overview
grouppuppet-users @
postedAug 8, '13 at 11:07a
activeAug 8, '13 at 4:52p



site design / logo © 2022 Grokbase