FAQ
Provide a command line option to check if a Hadoop jobtracker is idle
---------------------------------------------------------------------

Key: HADOOP-2266
URL: https://issues.apache.org/jira/browse/HADOOP-2266
Project: Hadoop
Issue Type: New Feature
Components: mapred
Reporter: Hemanth Yamijala


This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.

This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Hemanth Yamijala (JIRA) at Nov 23, 2007 at 5:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Hemanth Yamijala updated HADOOP-2266:
    -------------------------------------

    Fix Version/s: 0.16.0
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Owen O'Malley (JIRA) at Nov 25, 2007 at 5:56 am
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545237 ]

    Owen O'Malley commented on HADOOP-2266:
    ---------------------------------------

    Another approach to this would be to have an admin command for the job tracker that would have the job tracker stop accepting new jobs and shut itself down when the last job completed. That way the external process wouldn't need to continually poll the job tracker to see if it was done yet. Thoughts?
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Enis Soztutar (JIRA) at Nov 26, 2007 at 8:21 am
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545356 ]

    Enis Soztutar commented on HADOOP-2266:
    ---------------------------------------

    bq. Another approach to this would be to have an admin command for the job tracker that would have the job tracker stop accepting new jobs and shut itself down when the last job completed.
    I remember this was discussed before but never implemented. I think this and a command to gracefully stop the jt would be great.
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Michael Bieniosek (JIRA) at Nov 27, 2007 at 6:10 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545943 ]

    Michael Bieniosek commented on HADOOP-2266:
    -------------------------------------------

    It's possible to do this through the JobClient API currently.
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hemanth Yamijala (JIRA) at Nov 28, 2007 at 3:58 am
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546094 ]

    Hemanth Yamijala commented on HADOOP-2266:
    ------------------------------------------

    bq. Another approach to this would be to have an admin command for the job tracker that would have the job tracker stop accepting new jobs and shut itself down when the last job completed. That way the external process wouldn't need to continually poll the job tracker to see if it was done yet. Thoughts?

    The requirement we have in Hadoop On Demand is to shutdown a job tracker when it has not run any jobs for a period of time. This is to help in releasing unused resources, and reclaim them. The approach suggested above actually makes sense if we want to run the job tracker for a period of time and then shutdown. This seems to be different from what we need, right ?

    The problem of continous polling does exist. But I feel it may not be too intensive. Any alternative that does not have this problem, but solves the use-case would also be great.
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Nov 28, 2007 at 5:52 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546317 ]

    Arun C Murthy commented on HADOOP-2266:
    ---------------------------------------

    The other other issue is see with the JT shutting itself down is that the TTs (and HoD) don't know why that happened. If we just implement the *isidle <for_this_long>*, we could let HoD make the decision to shut down the entire cluster via the usual scripts... thoughts?
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Nov 28, 2007 at 5:58 pm
    [ https://issues.apache.org/jira/browse/HADOOP-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546320 ]

    Arun C Murthy commented on HADOOP-2266:
    ---------------------------------------

    Let me clarify, I agree 'graceful shutdown' initiated by the JT is the _right_ solution.

    However, the command, as described here, is a good interim solution IMHO.
    Provide a command line option to check if a Hadoop jobtracker is idle
    ---------------------------------------------------------------------

    Key: HADOOP-2266
    URL: https://issues.apache.org/jira/browse/HADOOP-2266
    Project: Hadoop
    Issue Type: New Feature
    Components: mapred
    Reporter: Hemanth Yamijala
    Fix For: 0.16.0


    This is an RFE for providing a way to determine from the hadoop command line whether a jobtracker is idle. One possibility is to have something like hadoop jobtracker -idle <time>. Hadoop would return true (maybe via some stdout output) if the jobtracker had no work to do (jobs running / prepared) since <time> seconds, false otherwise.
    This would be useful for management / provisioning systems like Hadoop-On-Demand [HADOOP-1301], which can then deallocate the idle, provisioned clusters automatically, and release resources.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedNov 23, '07 at 5:07a
activeNov 28, '07 at 5:58p
posts8
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Arun C Murthy (JIRA): 8 posts

People

Translate

site design / logo © 2022 Grokbase