FAQ
Hi all,

I recently started using the Nagios resources to populate a monitoring
server. I cycle hosts fairly quickly in my environment so already I've had
five hosts come and go from under Nagios. After a fair amount of searching
I discovered "puppet node deactivate" and performed same on the removed
hosts -- I'm still trying to figure out how to make that part of the
process going forward -- but somehow only one of my five hosts ended up
being removed from my Nagios configs. I am using resource { purge => true
} for the affected resource types.

So here's where I'm at -- if I run puppet node status <node> on any of
these missing hosts, they appear as "deactivated". If I clear out my
nagios config files and re-run puppet agent, the nodes and their services
(I did have an exported nagios_service resource when these hosts were
alive, which I've since removed -- in case that matters) will re-appear.
I've tried *puppet node clean*, *puppet node destroy*, *puppet node
deactivate*, with and without terminus=puppetdb. I can see in the puppetdb
log that it has received multiple deactivate commands for these hosts.
Nonetheless, the items are still appearing when the nagios host performs a
collection. I've got to put an end to that!

An example of puppet node status for one of the affected:

# puppet node status [badnodename]
[badnodename]
Deactivated at 2013-04-03T23:00:55.349Z
No catalog received
No facts received


One interesting bit. First, when I run puppet agent --test, during catalog
compilation I _was_ seeing the following for all four affected hosts:

warning: Nagios_service check_ssh_[badhost] found in both naginator and
naginator; skipping the naginator version


Out of frustration, I disabled storedconfigurations on the puppet master
and restarted it. After a few catalogs had processed, I updated Nagios
after manually purging the hosts file. As expected, all my host and
service definitions disappeared. I then re-enabled storedconfigurations,
fingers-crossed that the old host defs would be gone -- to my dismay, they
reappeared on the *second* catalog run after I brought stored_configs back
online. Now, I no longer get the error message above, but I do still get
the deactivated hosts and associated services.

At this point, I'd be happy to simply wipe all of my existing stored
configurations, as I suspect these four hosts have gotten themselves into
some kind of limbo. However, while the instructions for removing stored
config data from the old MySQL puppet db is quite straightforward, it
doesn't seem to map to the postgresql DB schema, and I can't find any
advice on how to go about wiping it there. I'd prefer not to just scrap my
database, but I'm getting closer to that point.

Any help would be appreciated.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Michael O'Dea at Apr 4, 2013 at 8:08 pm
    < exasperated sigh >

    So, the hosts in question have, as part of their unique hostnames, a small
    hex string which is in uppercase. In the Nagios view where I was seeing
    these offline hosts, the hostname's case was preserved. Within PuppetDB
    however, the hostname is all lowercase. As a consequence, these hosts were
    not removed because instead of deactivating that host, a *new* certname
    entry was created, with my mixed-case value, and *that* entry was marked as
    deactivated. The actual host was still humming right along.

    Well, I guess I learned something. Technically there's a bug there -- but
    I've lost a whole day on this issue so I'm going to refrain from reporting
    it at the moment. Hope this helps someone else. If you're deactivating a
    host with a capital letter, odds are good you're not deactivating what you
    think you're deactivating.

    On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O'Dea wrote:

    Hi all,

    I recently started using the Nagios resources to populate a monitoring
    server. I cycle hosts fairly quickly in my environment so already I've had
    five hosts come and go from under Nagios. After a fair amount of searching
    I discovered "puppet node deactivate" and performed same on the removed
    hosts -- I'm still trying to figure out how to make that part of the
    process going forward -- but somehow only one of my five hosts ended up
    being removed from my Nagios configs. I am using resource { purge => true
    } for the affected resource types.

    So here's where I'm at -- if I run puppet node status <node> on any of
    these missing hosts, they appear as "deactivated". If I clear out my
    nagios config files and re-run puppet agent, the nodes and their services
    (I did have an exported nagios_service resource when these hosts were
    alive, which I've since removed -- in case that matters) will re-appear.
    I've tried *puppet node clean*, *puppet node destroy*, *puppet node
    deactivate*, with and without terminus=puppetdb. I can see in the
    puppetdb log that it has received multiple deactivate commands for these
    hosts. Nonetheless, the items are still appearing when the nagios host
    performs a collection. I've got to put an end to that!

    An example of puppet node status for one of the affected:

    # puppet node status [badnodename]
    [badnodename]
    Deactivated at 2013-04-03T23:00:55.349Z
    No catalog received
    No facts received


    One interesting bit. First, when I run puppet agent --test, during
    catalog compilation I _was_ seeing the following for all four affected
    hosts:

    warning: Nagios_service check_ssh_[badhost] found in both naginator and
    naginator; skipping the naginator version


    Out of frustration, I disabled storedconfigurations on the puppet master
    and restarted it. After a few catalogs had processed, I updated Nagios
    after manually purging the hosts file. As expected, all my host and
    service definitions disappeared. I then re-enabled storedconfigurations,
    fingers-crossed that the old host defs would be gone -- to my dismay, they
    reappeared on the *second* catalog run after I brought stored_configs back
    online. Now, I no longer get the error message above, but I do still get
    the deactivated hosts and associated services.

    At this point, I'd be happy to simply wipe all of my existing stored
    configurations, as I suspect these four hosts have gotten themselves into
    some kind of limbo. However, while the instructions for removing stored
    config data from the old MySQL puppet db is quite straightforward, it
    doesn't seem to map to the postgresql DB schema, and I can't find any
    advice on how to go about wiping it there. I'd prefer not to just scrap my
    database, but I'm getting closer to that point.

    Any help would be appreciated.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Ken Barber at Apr 5, 2013 at 12:12 am
    Yeah, sounds like a bug Michael or at least something we can improve
    upon - DNS is case insensitive: https://tools.ietf.org/rfc/rfc4343.txt
    - but to make things more complex you can override what the node name
    is in Puppet (ie. node_name_fact and node_name_value), so I can only
    imagine the fun debates on such an issue :-).

    Raise a bug though, and we'll ponder the problem:

    https://projects.puppetlabs.com/projects/puppetdb/issues/new

    ken.
    On Thu, Apr 4, 2013 at 9:08 PM, Michael O'Dea wrote:
    < exasperated sigh >

    So, the hosts in question have, as part of their unique hostnames, a small
    hex string which is in uppercase. In the Nagios view where I was seeing
    these offline hosts, the hostname's case was preserved. Within PuppetDB
    however, the hostname is all lowercase. As a consequence, these hosts were
    not removed because instead of deactivating that host, a new certname entry
    was created, with my mixed-case value, and that entry was marked as
    deactivated. The actual host was still humming right along.

    Well, I guess I learned something. Technically there's a bug there -- but
    I've lost a whole day on this issue so I'm going to refrain from reporting
    it at the moment. Hope this helps someone else. If you're deactivating a
    host with a capital letter, odds are good you're not deactivating what you
    think you're deactivating.

    On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O'Dea wrote:

    Hi all,

    I recently started using the Nagios resources to populate a monitoring
    server. I cycle hosts fairly quickly in my environment so already I've had
    five hosts come and go from under Nagios. After a fair amount of searching
    I discovered "puppet node deactivate" and performed same on the removed
    hosts -- I'm still trying to figure out how to make that part of the process
    going forward -- but somehow only one of my five hosts ended up being
    removed from my Nagios configs. I am using resource { purge => true } for
    the affected resource types.

    So here's where I'm at -- if I run puppet node status <node> on any of
    these missing hosts, they appear as "deactivated". If I clear out my nagios
    config files and re-run puppet agent, the nodes and their services (I did
    have an exported nagios_service resource when these hosts were alive, which
    I've since removed -- in case that matters) will re-appear. I've tried
    puppet node clean, puppet node destroy, puppet node deactivate, with and
    without terminus=puppetdb. I can see in the puppetdb log that it has
    received multiple deactivate commands for these hosts. Nonetheless, the
    items are still appearing when the nagios host performs a collection. I've
    got to put an end to that!

    An example of puppet node status for one of the affected:

    # puppet node status [badnodename]
    [badnodename]
    Deactivated at 2013-04-03T23:00:55.349Z
    No catalog received
    No facts received


    One interesting bit. First, when I run puppet agent --test, during
    catalog compilation I _was_ seeing the following for all four affected
    hosts:

    warning: Nagios_service check_ssh_[badhost] found in both naginator and
    naginator; skipping the naginator version


    Out of frustration, I disabled storedconfigurations on the puppet master
    and restarted it. After a few catalogs had processed, I updated Nagios
    after manually purging the hosts file. As expected, all my host and service
    definitions disappeared. I then re-enabled storedconfigurations,
    fingers-crossed that the old host defs would be gone -- to my dismay, they
    reappeared on the *second* catalog run after I brought stored_configs back
    online. Now, I no longer get the error message above, but I do still get
    the deactivated hosts and associated services.

    At this point, I'd be happy to simply wipe all of my existing stored
    configurations, as I suspect these four hosts have gotten themselves into
    some kind of limbo. However, while the instructions for removing stored
    config data from the old MySQL puppet db is quite straightforward, it
    doesn't seem to map to the postgresql DB schema, and I can't find any advice
    on how to go about wiping it there. I'd prefer not to just scrap my
    database, but I'm getting closer to that point.

    Any help would be appreciated.
    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Nick Lewis at Apr 5, 2013 at 2:27 am

    On Thursday, April 4, 2013 1:08:06 PM UTC-7, Michael O'Dea wrote:

    < exasperated sigh >

    So, the hosts in question have, as part of their unique hostnames, a small
    hex string which is in uppercase. In the Nagios view where I was seeing
    these offline hosts, the hostname's case was preserved. Within PuppetDB
    however, the hostname is all lowercase. As a consequence, these hosts were
    not removed because instead of deactivating that host, a *new* certname
    entry was created, with my mixed-case value, and *that* entry was marked
    as deactivated. The actual host was still humming right along.

    Well, I guess I learned something. Technically there's a bug there -- but
    I've lost a whole day on this issue so I'm going to refrain from reporting
    it at the moment. Hope this helps someone else. If you're deactivating a
    host with a capital letter, odds are good you're not deactivating what you
    think you're deactivating.
    I'll join in that exasperated sigh with you. :) This issue pops up fairly
    commonly (albeit usually with hostname vs. fqdn). I remember a conversation
    on irc about having the `puppet node deactivate` command first check if the
    node exists. It could then fail if the node doesn't (because you probably
    just got the name wrong), or have an option to force (if for some reason
    you want to proactively tell PuppetDB a node exists, but is inactive).
    Potentially, we could also perform a regex query and make suggestions. I'm
    not sure what happened with that idea, but I guess it didn't go anywhere.
    At any rate, I think the frustration of accidentally deactivating the wrong
    nodes far outweighs whatever rationale someone might have for deactivating
    a node that doesn't exist. This needs to change.

    I'm sorry for your trouble. I'll make sure to follow up on making that
    change, this time.

    On Thursday, April 4, 2013 3:19:00 PM UTC-4, Michael O'Dea wrote:

    Hi all,

    I recently started using the Nagios resources to populate a monitoring
    server. I cycle hosts fairly quickly in my environment so already I've had
    five hosts come and go from under Nagios. After a fair amount of searching
    I discovered "puppet node deactivate" and performed same on the removed
    hosts -- I'm still trying to figure out how to make that part of the
    process going forward -- but somehow only one of my five hosts ended up
    being removed from my Nagios configs. I am using resource { purge => true
    } for the affected resource types.

    So here's where I'm at -- if I run puppet node status <node> on any of
    these missing hosts, they appear as "deactivated". If I clear out my
    nagios config files and re-run puppet agent, the nodes and their services
    (I did have an exported nagios_service resource when these hosts were
    alive, which I've since removed -- in case that matters) will re-appear.
    I've tried *puppet node clean*, *puppet node destroy*, *puppet node
    deactivate*, with and without terminus=puppetdb. I can see in the
    puppetdb log that it has received multiple deactivate commands for these
    hosts. Nonetheless, the items are still appearing when the nagios host
    performs a collection. I've got to put an end to that!

    An example of puppet node status for one of the affected:

    # puppet node status [badnodename]
    [badnodename]
    Deactivated at 2013-04-03T23:00:55.349Z
    No catalog received
    No facts received


    One interesting bit. First, when I run puppet agent --test, during
    catalog compilation I _was_ seeing the following for all four affected
    hosts:

    warning: Nagios_service check_ssh_[badhost] found in both naginator and
    naginator; skipping the naginator version


    Out of frustration, I disabled storedconfigurations on the puppet master
    and restarted it. After a few catalogs had processed, I updated Nagios
    after manually purging the hosts file. As expected, all my host and
    service definitions disappeared. I then re-enabled storedconfigurations,
    fingers-crossed that the old host defs would be gone -- to my dismay, they
    reappeared on the *second* catalog run after I brought stored_configs back
    online. Now, I no longer get the error message above, but I do still get
    the deactivated hosts and associated services.

    At this point, I'd be happy to simply wipe all of my existing stored
    configurations, as I suspect these four hosts have gotten themselves into
    some kind of limbo. However, while the instructions for removing stored
    config data from the old MySQL puppet db is quite straightforward, it
    doesn't seem to map to the postgresql DB schema, and I can't find any
    advice on how to go about wiping it there. I'd prefer not to just scrap my
    database, but I'm getting closer to that point.

    Any help would be appreciated.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedApr 4, '13 at 7:19p
activeApr 5, '13 at 2:27a
posts4
users3
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase