FAQ
Hi all,

We've come across a rather strange problem where the parameters of some
resources in PuppetDB are now empty.

We have a Nagios server collecting resources from PuppetDB and we've
started to get failures like this for one resource type:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver

The Puppet manifest that defines that resources is this, it is impossible
to not populate host_alias:

*******************************
define nagios::host::host($host = $::fqdn, $tag = undef) {
   @@nagios::config::host { $host:
     host_alias => $host,
     address => $host,
     tag => $use_tag,
   }
}
*******************************

If we query PuppetDB directly (redacted), there are indeed no parameters at
all on this resource:

*******************************
# curl -H 'Accept: application/json' -X GET
'https://puppet:8081/v2/resources' --cacert
/var/lib/puppet/ssl/ca/ca_crt.pem --cert
/var/lib/puppet/ssl/certs/puppet.pem --key
/var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
"type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
[
   {
     "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
     "parameters": {
     },
     "title": "hostname",
     "exported": true,
     "certname": "hostname",
     "type": "Nagios::Config::Host",
     "sourceline": 27,
     "sourcefile":
"/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
     "tags": [
     ]
   }
]
*******************************

After a Puppet run and a new catalog, this resource now looks normal:

*******************************
# curl -H 'Accept: application/json' -X GET
'https://puppet:8081/v2/resources' --cacert
/var/lib/puppet/ssl/ca/ca_crt.pem --cert
/var/lib/puppet/ssl/certs/puppet.pem --key
/var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
"type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
[
   {
     "type": "Nagios::Config::Host",
     "sourceline": 27,
     "title": "hostname",
     "certname": "hostname",
     "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
     "parameters": {
       "address": "hostname",
       "tag": "tag",
       "host_alias": "hostname"
     },
     "exported": true,
     "sourcefile":
"/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
     "tags": [
     ]
   }
]
*******************************

These nodes do not have Puppet run on them regularly. We did upgrade from
PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We
don't do any automatic report or node expiry.

This started happening back on 2nd August, just halfway through the day the
Puppet runs on the Nagios server start failing with this error. Now if I
think back, at this time I think I had a broken Nagios module and a lot of
manifests were failing to compile, but I fixed this and re-ran the
failures, and everything was ok. PuppetDB only stores the last catalog, so
there's no way a broken catalog could have stayed there, right?

I've fixed this by refreshing the catalog of all nodes in PuppetDB, but
I've got no idea how it got into this state. Any ideas?

Thanks,

-Luke

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Ken Barber at Aug 8, 2013 at 11:48 am

    We've come across a rather strange problem where the parameters of some
    resources in PuppetDB are now empty.

    We have a Nagios server collecting resources from PuppetDB and we've started
    to get failures like this for one resource type:

    Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
    Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver

    The Puppet manifest that defines that resources is this, it is impossible to
    not populate host_alias:

    *******************************
    define nagios::host::host($host = $::fqdn, $tag = undef) {
    @@nagios::config::host { $host:
    host_alias => $host,
    address => $host,
    tag => $use_tag,
    }
    }
    *******************************

    If we query PuppetDB directly (redacted), there are indeed no parameters at
    all on this resource:

    *******************************
    # curl -H 'Accept: application/json' -X GET
    'https://puppet:8081/v2/resources' --cacert
    /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
    [
    {
    "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
    "parameters": {
    },
    "title": "hostname",
    "exported": true,
    "certname": "hostname",
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "tags": [
    ]
    }
    ]
    *******************************

    After a Puppet run and a new catalog, this resource now looks normal:

    *******************************
    # curl -H 'Accept: application/json' -X GET
    'https://puppet:8081/v2/resources' --cacert
    /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
    [
    {
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "title": "hostname",
    "certname": "hostname",
    "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
    "parameters": {
    "address": "hostname",
    "tag": "tag",
    "host_alias": "hostname"
    },
    "exported": true,
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "tags": [
    ]
    }
    ]
    *******************************

    These nodes do not have Puppet run on them regularly. We did upgrade from
    PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We
    don't do any automatic report or node expiry.

    This started happening back on 2nd August, just halfway through the day the
    Puppet runs on the Nagios server start failing with this error. Now if I
    think back, at this time I think I had a broken Nagios module and a lot of
    manifests were failing to compile, but I fixed this and re-ran the failures,
    and everything was ok. PuppetDB only stores the last catalog, so there's no
    way a broken catalog could have stayed there, right?

    I've fixed this by refreshing the catalog of all nodes in PuppetDB, but I've
    got no idea how it got into this state. Any ideas?
    No good idea yet, but there is something suspicious in your curl
    responses - the "resource" hash, did you obfuscate this yourself on
    purpose? The two hashes between the first and second requests are
    identical. That hash is calculated based on the sum of the resource,
    including parameters - so it seems impossible that PuppetDB arrived at
    the same hash with and without parameters.

    Maybe just maybe the responses were identical, and somehow PuppetDB
    was not returning parameters as a fault. This might indicate some sort
    of integrity problem, whereby the link to the parameters in the RDBMS
    was lost somehow, although this is the first time I've heard of it
    Luke. Maybe this was an upgrade schema change failure between 1.0.1
    and 1.3.2? I'd have to consult what changed in the schema between
    those two points to determine if this is likely however. Does the
    timing of your upgrade, and the first time you saw this fault line up
    with such a possibility? Remember a schema change will only occur
    after a restart of PuppetDB ... (so perhaps consult your logs to see
    when this happened after upgrade).

    Let me at least try to replicate while I await your responses.

    ken.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Aug 8, 2013 at 12:42 pm
    Well FWIW, I can't replicate this when I upgrade from 1.0.1 to 1.3.2,
    I even attempted to name things in the same way, no luck:

    https://gist.github.com/kbarber/6184213.

    Looking at the schema changes in code, not much has changed to do with
    parameters since 1.0.1 either:

    https://github.com/puppetlabs/puppetdb/blob/master/src/com/puppetlabs/puppetdb/scf/migrate.clj#L326-L331

    So its probably not specifically a bug with the schema change.
    On Thu, Aug 8, 2013 at 12:48 PM, Ken Barber wrote:
    We've come across a rather strange problem where the parameters of some
    resources in PuppetDB are now empty.

    We have a Nagios server collecting resources from PuppetDB and we've started
    to get failures like this for one resource type:

    Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
    Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver

    The Puppet manifest that defines that resources is this, it is impossible to
    not populate host_alias:

    *******************************
    define nagios::host::host($host = $::fqdn, $tag = undef) {
    @@nagios::config::host { $host:
    host_alias => $host,
    address => $host,
    tag => $use_tag,
    }
    }
    *******************************

    If we query PuppetDB directly (redacted), there are indeed no parameters at
    all on this resource:

    *******************************
    # curl -H 'Accept: application/json' -X GET
    'https://puppet:8081/v2/resources' --cacert
    /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
    [
    {
    "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
    "parameters": {
    },
    "title": "hostname",
    "exported": true,
    "certname": "hostname",
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "tags": [
    ]
    }
    ]
    *******************************

    After a Puppet run and a new catalog, this resource now looks normal:

    *******************************
    # curl -H 'Accept: application/json' -X GET
    'https://puppet:8081/v2/resources' --cacert
    /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
    [
    {
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "title": "hostname",
    "certname": "hostname",
    "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
    "parameters": {
    "address": "hostname",
    "tag": "tag",
    "host_alias": "hostname"
    },
    "exported": true,
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "tags": [
    ]
    }
    ]
    *******************************

    These nodes do not have Puppet run on them regularly. We did upgrade from
    PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We
    don't do any automatic report or node expiry.

    This started happening back on 2nd August, just halfway through the day the
    Puppet runs on the Nagios server start failing with this error. Now if I
    think back, at this time I think I had a broken Nagios module and a lot of
    manifests were failing to compile, but I fixed this and re-ran the failures,
    and everything was ok. PuppetDB only stores the last catalog, so there's no
    way a broken catalog could have stayed there, right?

    I've fixed this by refreshing the catalog of all nodes in PuppetDB, but I've
    got no idea how it got into this state. Any ideas?
    No good idea yet, but there is something suspicious in your curl
    responses - the "resource" hash, did you obfuscate this yourself on
    purpose? The two hashes between the first and second requests are
    identical. That hash is calculated based on the sum of the resource,
    including parameters - so it seems impossible that PuppetDB arrived at
    the same hash with and without parameters.

    Maybe just maybe the responses were identical, and somehow PuppetDB
    was not returning parameters as a fault. This might indicate some sort
    of integrity problem, whereby the link to the parameters in the RDBMS
    was lost somehow, although this is the first time I've heard of it
    Luke. Maybe this was an upgrade schema change failure between 1.0.1
    and 1.3.2? I'd have to consult what changed in the schema between
    those two points to determine if this is likely however. Does the
    timing of your upgrade, and the first time you saw this fault line up
    with such a possibility? Remember a schema change will only occur
    after a restart of PuppetDB ... (so perhaps consult your logs to see
    when this happened after upgrade).

    Let me at least try to replicate while I await your responses.

    ken.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Luke Bigum at Aug 8, 2013 at 1:03 pm

    On Thursday, August 8, 2013 12:48:03 PM UTC+1, Ken Barber wrote:

    No good idea yet, but there is something suspicious in your curl
    responses - the "resource" hash, did you obfuscate this yourself on
    purpose? The two hashes between the first and second requests are
    identical. That hash is calculated based on the sum of the resource,
    including parameters - so it seems impossible that PuppetDB arrived at
    the same hash with and without parameters.
    I think that's just me being too sensorship heavy and abusing copy and
    paste, I would have copied some fields from the same example. Trust me that
    the resources dictionary was empty though ;-)

    Maybe just maybe the responses were identical, and somehow PuppetDB
    was not returning parameters as a fault. This might indicate some sort
    of integrity problem, whereby the link to the parameters in the RDBMS
    was lost somehow, although this is the first time I've heard of it
    Luke. Maybe this was an upgrade schema change failure between 1.0.1
    and 1.3.2?

    We thought of that, however we upgraded 3 weeks ago and I saw working
    reports in my Dashboard on the Nagios server up until Friday last week, so
    it was able to collect valid resources for a long while after the DB
    migration. What happened on Friday though was I basically had this running:
    the database
    mco rpc shellout cmd cmd="puppet agent --test --noop" -T mcollective

    Then I realised we had broken manifests everywhere, so I fixed it and
    started MC again. From that point onward the Nagios server was unable to
    collect as it was getting these dodgy resources from PuppetDB. What doesn't
    make sense is my subsequent site wide no-op should have replaced every
    manifest in PuppetDB, so I'm stumped.

    I'd have to consult what changed in the schema between
    those two points to determine if this is likely however. Does the
    timing of your upgrade, and the first time you saw this fault line up
    with such a possibility? Remember a schema change will only occur
    after a restart of PuppetDB ... (so perhaps consult your logs to see
    when this happened after upgrade).
    Just checked - PuppetDB restarted at the same time the RPMs were upgraded
    and hasn't restarted since.

    Let me at least try to replicate while I await your responses.
    >

    Now if I was thinking smart I would have taken a Postgres backup before I
    re-freshed all the catalogs, but I didn't, not sure if that would have
    helped much. I agree with subsequent posts as well - probably not a
    migration problem.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Aug 8, 2013 at 1:14 pm

    No good idea yet, but there is something suspicious in your curl
    responses - the "resource" hash, did you obfuscate this yourself on
    purpose? The two hashes between the first and second requests are
    identical. That hash is calculated based on the sum of the resource,
    including parameters - so it seems impossible that PuppetDB arrived at
    the same hash with and without parameters.

    I think that's just me being too sensorship heavy and abusing copy and
    paste, I would have copied some fields from the same example. Trust me that
    the resources dictionary was empty though ;-)
    So just to clarify, the resources hash
    '8ba4379c364b9dba9d18836ef52ce5f4f82d0468' was different or the same
    between the two examples?
    Now if I was thinking smart I would have taken a Postgres backup before I
    re-freshed all the catalogs, but I didn't, not sure if that would have
    helped much. I agree with subsequent posts as well - probably not a
    migration problem.
    It might have helped. Are any other nodes and resources still
    exhibiting this strange behaviour? Maybe checking for any exported
    resources with no params might be worthwhile.

    ken.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Luke Bigum at Aug 8, 2013 at 2:20 pm

    On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote:
    I think that's just me being too sensorship heavy and abusing copy and
    paste, I would have copied some fields from the same example. Trust me that
    the resources dictionary was empty though ;-)
    So just to clarify, the resources hash
    '8ba4379c364b9dba9d18836ef52ce5f4f82d0468' was different or the same
    between the two examples?
    Actually they are the same, my copy and paste skills remain rock solid for
    another day.

    I found some more broken resources belonging to some dev servers with a
    handy jgrep:

    curl -H 'Accept: application/json' -X GET
    'https://puppet:8081/v2/resources' --cacert
    /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "parameters.host_alias=null"

    This is the hostname redacted JSON before:

    ***************************
    [
       {
         "type": "Nagios::Config::Host",
         "tags": [
           "nagios::host",
           "default",
           "node",
           "config",
           "nagios::config::host",
           "hostname",
           "en1",
           "host",
           "nagios::host::host",
           "undef",
           "class",
           "nagios"
         ],
         "parameters": {
         },
         "certname": "hostname",
         "title": "hostname",
         "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
         "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
         "sourceline": 27,
         "exported": true
       }
    ]
    ***************************

    And after a Puppet run to refresh the catalog:


    ***************************
    [
       {
         "type": "Nagios::Config::Host",
         "sourceline": 27,
         "certname": "hostname",
         "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
         "exported": true,
         "title": "hostname",
         "tags": [
           "nagios::host",
           "node",
           "config",
           "nagios::config::host",
           "hostname",
           "en1",
           "host",
           "nagios::host::host",
           "undef",
           "base",
           "class",
           "nagios"
         ],
         "parameters": {
           "host_alias": "hostname",
           "tag": "en1",
           "address": "hostname"
         },
         "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp"
       }
    ]
    ***************************

    So there's 12 resources with this problem remaining now.

    -Luke
    Now if I was thinking smart I would have taken a Postgres backup before I
    re-freshed all the catalogs, but I didn't, not sure if that would have
    helped much. I agree with subsequent posts as well - probably not a
    migration problem.
    It might have helped. Are any other nodes and resources still
    exhibiting this strange behaviour? Maybe checking for any exported
    resources with no params might be worthwhile.

    ken.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Aug 8, 2013 at 2:23 pm
    Wow. Good, don't do anything ... I'll reach out to you directly.
    On Thu, Aug 8, 2013 at 3:20 PM, Luke Bigum wrote:

    On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote:

    I think that's just me being too sensorship heavy and abusing copy and
    paste, I would have copied some fields from the same example. Trust me
    that
    the resources dictionary was empty though ;-)
    So just to clarify, the resources hash
    '8ba4379c364b9dba9d18836ef52ce5f4f82d0468' was different or the same
    between the two examples?

    Actually they are the same, my copy and paste skills remain rock solid for
    another day.

    I found some more broken resources belonging to some dev servers with a
    handy jgrep:

    curl -H 'Accept: application/json' -X GET 'https://puppet:8081/v2/resources'
    --cacert /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "parameters.host_alias=null"

    This is the hostname redacted JSON before:

    ***************************
    [
    {
    "type": "Nagios::Config::Host",
    "tags": [
    "nagios::host",
    "default",
    "node",
    "config",
    "nagios::config::host",
    "hostname",
    "en1",
    "host",
    "nagios::host::host",
    "undef",
    "class",
    "nagios"
    ],
    "parameters": {
    },
    "certname": "hostname",
    "title": "hostname",
    "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "sourceline": 27,
    "exported": true
    }
    ]
    ***************************

    And after a Puppet run to refresh the catalog:


    ***************************
    [
    {
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "certname": "hostname",
    "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
    "exported": true,
    "title": "hostname",
    "tags": [
    "nagios::host",
    "node",
    "config",
    "nagios::config::host",
    "hostname",
    "en1",
    "host",
    "nagios::host::host",
    "undef",
    "base",
    "class",
    "nagios"
    ],
    "parameters": {
    "host_alias": "hostname",
    "tag": "en1",
    "address": "hostname"
    },
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp"
    }
    ]
    ***************************

    So there's 12 resources with this problem remaining now.

    -Luke
    Now if I was thinking smart I would have taken a Postgres backup before
    I
    re-freshed all the catalogs, but I didn't, not sure if that would have
    helped much. I agree with subsequent posts as well - probably not a
    migration problem.
    It might have helped. Are any other nodes and resources still
    exhibiting this strange behaviour? Maybe checking for any exported
    resources with no params might be worthwhile.

    ken.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Aug 8, 2013 at 4:52 pm
    For those following along at home, Luke and I continued to dig on IRC
    ... he was actually using HSQLDB not Postgresql it turns out. Doing
    queries on the database resulted in missing rows from resource_params
    for some reason, which a puppet run restored. We can't find a
    reasonable explanation for this yet, and are still looking. FWIW, this
    is the first time I've seen this behaviour.
    On Thu, Aug 8, 2013 at 3:22 PM, Ken Barber wrote:
    Wow. Good, don't do anything ... I'll reach out to you directly.
    On Thu, Aug 8, 2013 at 3:20 PM, Luke Bigum wrote:

    On Thursday, August 8, 2013 2:14:33 PM UTC+1, Ken Barber wrote:

    I think that's just me being too sensorship heavy and abusing copy and
    paste, I would have copied some fields from the same example. Trust me
    that
    the resources dictionary was empty though ;-)
    So just to clarify, the resources hash
    '8ba4379c364b9dba9d18836ef52ce5f4f82d0468' was different or the same
    between the two examples?

    Actually they are the same, my copy and paste skills remain rock solid for
    another day.

    I found some more broken resources belonging to some dev servers with a
    handy jgrep:

    curl -H 'Accept: application/json' -X GET 'https://puppet:8081/v2/resources'
    --cacert /var/lib/puppet/ssl/ca/ca_crt.pem --cert
    /var/lib/puppet/ssl/certs/puppet.pem --key
    /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
    "type", "Nagios::Config::Host"]' | jgrep "parameters.host_alias=null"

    This is the hostname redacted JSON before:

    ***************************
    [
    {
    "type": "Nagios::Config::Host",
    "tags": [
    "nagios::host",
    "default",
    "node",
    "config",
    "nagios::config::host",
    "hostname",
    "en1",
    "host",
    "nagios::host::host",
    "undef",
    "class",
    "nagios"
    ],
    "parameters": {
    },
    "certname": "hostname",
    "title": "hostname",
    "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
    "sourceline": 27,
    "exported": true
    }
    ]
    ***************************

    And after a Puppet run to refresh the catalog:


    ***************************
    [
    {
    "type": "Nagios::Config::Host",
    "sourceline": 27,
    "certname": "hostname",
    "resource": "3368824b20c1eb7052952f574bb5547ca0c95a50",
    "exported": true,
    "title": "hostname",
    "tags": [
    "nagios::host",
    "node",
    "config",
    "nagios::config::host",
    "hostname",
    "en1",
    "host",
    "nagios::host::host",
    "undef",
    "base",
    "class",
    "nagios"
    ],
    "parameters": {
    "host_alias": "hostname",
    "tag": "en1",
    "address": "hostname"
    },
    "sourcefile":
    "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp"
    }
    ]
    ***************************

    So there's 12 resources with this problem remaining now.

    -Luke
    Now if I was thinking smart I would have taken a Postgres backup before
    I
    re-freshed all the catalogs, but I didn't, not sure if that would have
    helped much. I agree with subsequent posts as well - probably not a
    migration problem.
    It might have helped. Are any other nodes and resources still
    exhibiting this strange behaviour? Maybe checking for any exported
    resources with no params might be worthwhile.

    ken.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nicolai Mollerup at Aug 8, 2013 at 11:57 am
    Since you upgraded from 1.0 to 1.3.2 something has indeed happened in
    puppetdb so check your settings:

    http://docs.puppetlabs.com/puppetdb/1.3/release_notes.html#section-3

    This talks about automatic node purging, which will deactivate exported
    resources aswell.

    /Nicolai

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedAug 8, '13 at 11:07a
activeAug 8, '13 at 4:52p
posts9
users3
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase