FAQ
Hi Guys,

I am a new puppet user and wanted some type of monitoring for puppet so
deployed puppet-dashboard. It has been working very well for a few days
not, but all of a sudden I start getting pending tasks and they never
finish even after restarting all processes. They keep accumulating and
never seem to finish even though the clients are running fine. I have the
puppet-dashboard running on a different server than my puppetmaster. Is
there any way to troubleshoot what seems to be stuck and how to I get it
going without having to destroy the dashboard mysql db and recreating it
again? That is the only way I have got it working again but that is not
the correct way of doing this :) Any help would be appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Jo Rhett at Feb 9, 2013 at 1:25 am
    Make sure that your dashboard workers are running. If you are using CentOS then "service puppet-dashboard-workers status" will tell you.
    On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:
    I am a new puppet user and wanted some type of monitoring for puppet so deployed puppet-dashboard. It has been working very well for a few days not, but all of a sudden I start getting pending tasks and they never finish even after restarting all processes. They keep accumulating and never seem to finish even though the clients are running fine. I have the puppet-dashboard running on a different server than my puppetmaster. Is there any way to troubleshoot what seems to be stuck and how to I get it going without having to destroy the dashboard mysql db and recreating it again? That is the only way I have got it working again but that is not the correct way of doing this :) Any help would be appreciated.

    Thanks!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    Jo Rhett
    Net Consonance : net philanthropy to improve open source and internet projects.



    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jagga Soorma at Feb 9, 2013 at 2:44 am
    Yes, the workers are definitely running and I have tried restarting
    puppet-dashboard-workers a few times as well as rebooting the vm completely
    without any luck. Something seems to be stuck somewhere but I can't figure
    out what. This has happened a few times now and I have just been
    recreating from scratch to get it functional again. Are there any logs for
    the workers that might tell us why these pending jobs are stuck or not
    finishing?

    Thanks.
    On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett wrote:

    Make sure that your dashboard workers are running. If you are using CentOS
    then "service puppet-dashboard-workers status" will tell you.

    On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

    I am a new puppet user and wanted some type of monitoring for puppet so
    deployed puppet-dashboard. It has been working very well for a few days
    not, but all of a sudden I start getting pending tasks and they never
    finish even after restarting all processes. They keep accumulating and
    never seem to finish even though the clients are running fine. I have the
    puppet-dashboard running on a different server than my puppetmaster. Is
    there any way to troubleshoot what seems to be stuck and how to I get it
    going without having to destroy the dashboard mysql db and recreating it
    again? That is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.




    --
    Jo Rhett
    Net Consonance : net philanthropy to improve open source and internet
    projects.



    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jagga Soorma at Feb 10, 2013 at 3:11 am
    Anyone out there run into this issue before or have any idea about how to
    troubleshoot such a problem? Any help would be appreciated. I can't keep
    rebuilding puppet dashboard every time this happens.

    Thanks
    On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:

    Yes, the workers are definitely running and I have tried restarting
    puppet-dashboard-workers a few times as well as rebooting the vm completely
    without any luck. Something seems to be stuck somewhere but I can't figure
    out what. This has happened a few times now and I have just been
    recreating from scratch to get it functional again. Are there any logs for
    the workers that might tell us why these pending jobs are stuck or not
    finishing?

    Thanks.
    On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett wrote:

    Make sure that your dashboard workers are running. If you are using
    CentOS then "service puppet-dashboard-workers status" will tell you.

    On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

    I am a new puppet user and wanted some type of monitoring for puppet so
    deployed puppet-dashboard. It has been working very well for a few days
    not, but all of a sudden I start getting pending tasks and they never
    finish even after restarting all processes. They keep accumulating and
    never seem to finish even though the clients are running fine. I have the
    puppet-dashboard running on a different server than my puppetmaster. Is
    there any way to troubleshoot what seems to be stuck and how to I get it
    going without having to destroy the dashboard mysql db and recreating it
    again? That is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!

    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.




    --
    Jo Rhett
    Net Consonance : net philanthropy to improve open source and internet
    projects.



    --
    You received this message because you are subscribed to the Google Groups
    "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Esther Hanko at Feb 10, 2013 at 10:26 am
    Try looking in /usr/share/puppet-dashboard/log/.
    Did you update the dashboard or puppet master recently? I have ran into
    similar problems after updates, since I run the dashboard on a different
    port than the external_nodes script uses. The DASHBOARD_URL in
    /usr/share/puppet-dashboard/bin/external_node got changed back into
    localhost:3000 and as a result, all my nodes were seen as unresponsive.
    On 02/10/2013 04:11 AM, Jagga Soorma wrote:
    Anyone out there run into this issue before or have any idea about how
    to troubleshoot such a problem? Any help would be appreciated. I
    can't keep rebuilding puppet dashboard every time this happens.

    Thanks

    On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:

    Yes, the workers are definitely running and I have tried
    restarting puppet-dashboard-workers a few times as well as
    rebooting the vm completely without any luck. Something seems to
    be stuck somewhere but I can't figure out what. This has happened
    a few times now and I have just been recreating from scratch to
    get it functional again. Are there any logs for the workers that
    might tell us why these pending jobs are stuck or not finishing?

    Thanks.

    On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett wrote:

    Make sure that your dashboard workers are running. If you are
    using CentOS then "service puppet-dashboard-workers status"
    will tell you.
    On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:
    I am a new puppet user and wanted some type of monitoring for
    puppet so deployed puppet-dashboard. It has been working
    very well for a few days not, but all of a sudden I start
    getting pending tasks and they never finish even after
    restarting all processes. They keep accumulating and never
    seem to finish even though the clients are running fine. I
    have the puppet-dashboard running on a different server than
    my puppetmaster. Is there any way to troubleshoot what seems
    to be stuck and how to I get it going without having to
    destroy the dashboard mysql db and recreating it again? That
    is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!

    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to
    puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en
    <http://groups.google.com/group/puppet-users?hl=en>.
    For more options, visit
    https://groups.google.com/groups/opt_out
    <https://groups.google.com/groups/opt_out>.
    --
    Jo Rhett
    Net Consonance : net philanthropy to improve open source and
    internet projects.



    --
    You received this message because you are subscribed to the
    Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from
    it, send an email to puppet-users+unsubscribe@googlegroups.com
    To post to this group, send email to
    puppet-users@googlegroups.com
    Visit this group at
    http://groups.google.com/group/puppet-users?hl=en
    <http://groups.google.com/group/puppet-users?hl=en>.
    For more options, visit
    https://groups.google.com/groups/opt_out
    <https://groups.google.com/groups/opt_out>.



    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Felipe Salum at Feb 10, 2013 at 5:57 pm
    Are you having the problem after running 'puppet agent --test' a few times
    or it just happens without any manual run ?

    I was having the same problem some days ago, in my case because I was
    running 'puppet agent --test', more specifically the --show_diff that is
    used in --test was automatically generating a lot of output and causing the
    puppet dashboard workers to fail and increase the processing queue.

    Look at /usr/share/puppet-dashboard/log/delayed_job.log (or whatever your
    dashboard location is) and check what error it is giving.

    You can clean the queue manually without reinstalling everything:

    service puppet-dashboard-workers stop
    cd /usr/share/puppet-dashboard
    rm -f spool/*
    rake jobs:clear RAILS_ENV=production
    service puppet-dashboard-workers start

    Regards,
    Felipe
    On Saturday, February 9, 2013 7:11:04 PM UTC-8, Jagga Soorma wrote:

    Anyone out there run into this issue before or have any idea about how to
    troubleshoot such a problem? Any help would be appreciated. I can't keep
    rebuilding puppet dashboard every time this happens.

    Thanks
    On Friday, February 8, 2013 2:37:55 PM UTC-8, Jagga Soorma wrote:

    Yes, the workers are definitely running and I have tried restarting
    puppet-dashboard-workers a few times as well as rebooting the vm completely
    without any luck. Something seems to be stuck somewhere but I can't figure
    out what. This has happened a few times now and I have just been
    recreating from scratch to get it functional again. Are there any logs for
    the workers that might tell us why these pending jobs are stuck or not
    finishing?

    Thanks.

    On Fri, Feb 8, 2013 at 2:27 PM, Jo Rhett <jrh...@netconsonance.com<javascript:>
    wrote:
    Make sure that your dashboard workers are running. If you are using
    CentOS then "service puppet-dashboard-workers status" will tell you.

    On Feb 8, 2013, at 1:53 PM, Jagga Soorma wrote:

    I am a new puppet user and wanted some type of monitoring for puppet so
    deployed puppet-dashboard. It has been working very well for a few days
    not, but all of a sudden I start getting pending tasks and they never
    finish even after restarting all processes. They keep accumulating and
    never seem to finish even though the clients are running fine. I have the
    puppet-dashboard running on a different server than my puppetmaster. Is
    there any way to troubleshoot what seems to be stuck and how to I get it
    going without having to destroy the dashboard mysql db and recreating it
    again? That is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!

    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users...@googlegroups.com <javascript:>.
    To post to this group, send email to puppet...@googlegroups.com<javascript:>
    .
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.




    --
    Jo Rhett
    Net Consonance : net philanthropy to improve open source and internet
    projects.



    --
    You received this message because you are subscribed to the Google
    Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to puppet-users...@googlegroups.com <javascript:>.
    To post to this group, send email to puppet...@googlegroups.com<javascript:>
    .
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jagga Soorma at Feb 10, 2013 at 6:56 pm
    Thanks for pointing me to the log location. Looks like this might be the
    cause of the problem:

    --
    ..snip
    2013-02-10T10:40:35-0800: [Worker(delayed_job.3 host:ssfpcond01 pid:25048)]
    Report.create_from_yaml_file failed with ActiveRecord::StatementInvalid: *Mysql::Error:
    Data too long for column 'details' at row 1: INSERT INTO
    `delayed_job_failures` *(`summary`, `read`, `updated_at`, `backtrace`,
    `details`, `created_at`) VALUES('Importing report report-18121-1179.
    yaml', 0, '2013-02-10 18:40:35', '--- \n-
    /usr/share/puppet-dashboard/vendor/rails/activerecord/lib/active_record/connection_adapters/a
    ..snip..
    --

    I am managing my passwd & group file via puppet which at times have huge
    amount of changes especially in the group file. This could possibly be
    causing this error. Is there a way to increase the size of the details
    column to enable it to accept larger data?

    Has anyone seen this issue before?

    Thanks again for your help with this.
    On Friday, February 8, 2013 1:53:02 PM UTC-8, Jagga Soorma wrote:

    Hi Guys,

    I am a new puppet user and wanted some type of monitoring for puppet so
    deployed puppet-dashboard. It has been working very well for a few days
    not, but all of a sudden I start getting pending tasks and they never
    finish even after restarting all processes. They keep accumulating and
    never seem to finish even though the clients are running fine. I have the
    puppet-dashboard running on a different server than my puppetmaster. Is
    there any way to troubleshoot what seems to be stuck and how to I get it
    going without having to destroy the dashboard mysql db and recreating it
    again? That is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jagga Soorma at Feb 10, 2013 at 7:44 pm
    Looks like that was actually the problem. Searched google and came across
    the fix. Had to change the type of the messages column to mediumtext and
    the pending jobs started finishing. Did this by:

    alter table report_logs change column message message VARCHAR(65536);

    Just in case someone runs into this issue. Thanks.
    On Sunday, February 10, 2013 10:56:14 AM UTC-8, Jagga Soorma wrote:

    Thanks for pointing me to the log location. Looks like this might be the
    cause of the problem:

    --
    ..snip
    2013-02-10T10:40:35-0800: [Worker(delayed_job.3 host:ssfpcond01
    pid:25048)] Report.create_from_yaml_file failed with
    ActiveRecord::StatementInvalid: *Mysql::Error: Data too long for column
    'details' at row 1: INSERT INTO `delayed_job_failures` *(`summary`,
    `read`, `updated_at`, `backtrace`, `details`, `created_at`)
    VALUES('Importing report report-18121-1179.
    yaml', 0, '2013-02-10 18:40:35', '--- \n-
    /usr/share/puppet-dashboard/vendor/rails/activerecord/lib/active_record/connection_adapters/a
    ..snip..
    --

    I am managing my passwd & group file via puppet which at times have huge
    amount of changes especially in the group file. This could possibly be
    causing this error. Is there a way to increase the size of the details
    column to enable it to accept larger data?

    Has anyone seen this issue before?

    Thanks again for your help with this.
    On Friday, February 8, 2013 1:53:02 PM UTC-8, Jagga Soorma wrote:

    Hi Guys,

    I am a new puppet user and wanted some type of monitoring for puppet so
    deployed puppet-dashboard. It has been working very well for a few days
    not, but all of a sudden I start getting pending tasks and they never
    finish even after restarting all processes. They keep accumulating and
    never seem to finish even though the clients are running fine. I have the
    puppet-dashboard running on a different server than my puppetmaster. Is
    there any way to troubleshoot what seems to be stuck and how to I get it
    going without having to destroy the dashboard mysql db and recreating it
    again? That is the only way I have got it working again but that is not
    the correct way of doing this :) Any help would be appreciated.

    Thanks!
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Jesse Martinich at Jun 5, 2013 at 7:40 pm
    Your 'alter table' fix worked for me too. Thanks!

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users?hl=en.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Sean Penticoff at Sep 6, 2013 at 8:21 pm
    I ran into this as a result of the output of failed deps on a yum
    install and had a couple jobs that just wouldn't go away. After fixing
    the condition creating the error I used /usr/bin/rake jobs:clear to
    clear out the delayed_job queue and everything was back to normal.

    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Cko at Oct 11, 2013 at 11:07 am
    That fixed the problem for me. Thanks.
    On Friday, September 6, 2013 10:21:12 PM UTC+2, Psyber wrote:

    I ran into this as a result of the output of failed deps on a yum
    install and had a couple jobs that just wouldn't go away. After fixing
    the condition creating the error I used /usr/bin/rake jobs:clear to
    clear out the delayed_job queue and everything was back to normal.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Julien Nephtali at Oct 21, 2013 at 5:12 pm
    I found the solution to my problem here:
    http://projects.puppetlabs.com/issues/18411

    ALTER TABLE delayed_job_failures MODIFY details BLOB;

    that resolved the error.
    On Friday, October 11, 2013 7:07:02 AM UTC-4, cko wrote:

    That fixed the problem for me. Thanks.
    On Friday, September 6, 2013 10:21:12 PM UTC+2, Psyber wrote:

    I ran into this as a result of the output of failed deps on a yum
    install and had a couple jobs that just wouldn't go away. After fixing
    the condition creating the error I used /usr/bin/rake jobs:clear to
    clear out the delayed_job queue and everything was back to normal.
    --
    You received this message because you are subscribed to the Google Groups "Puppet Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
    To post to this group, send email to puppet-users@googlegroups.com.
    Visit this group at http://groups.google.com/group/puppet-users.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppuppet-users @
categoriespuppet
postedFeb 8, '13 at 9:53p
activeOct 21, '13 at 5:12p
posts12
users8
websitepuppetlabs.com

People

Translate

site design / logo © 2022 Grokbase