Grokbase › Groups › Hadoop › common-dev › August 2008
FAQ
Task tracker should not ask for tasks if its temp disk space is almost full
---------------------------------------------------------------------------

Key: HADOOP-4033
URL: https://issues.apache.org/jira/browse/HADOOP-4033
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.17.1
Reporter: Runping Qi



I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
Consequently, it had hard time to finish the tasks.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Runping Qi (JIRA) at Aug 27, 2008 at 1:31 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625939#action_12625939 ]

    Runping Qi commented on HADOOP-4033:
    ------------------------------------


    In the case where multiple temp dirs are configured, hadoop should avoid using a dir when it's available space is below certain threshold.

    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Ravi Gummadi (JIRA) at Jan 6, 2009 at 7:54 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Ravi Gummadi reassigned HADOOP-4033:
    ------------------------------------

    Assignee: Ravi Gummadi
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Ravi Gummadi (JIRA) at Jan 21, 2009 at 10:41 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665787#action_12665787 ]

    Ravi Gummadi commented on HADOOP-4033:
    --------------------------------------

    Configuring mapred.local.dir.minspacestart by user should solve this problem.

    Currently, the default value of mapred.local.dir.minspacestart is 0 bytes. How about changing this default value to a +ve value(say 1MB) in mapred.default.xml ?

    Thoughts ?
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Sameer Paranjpye (JIRA) at Jan 30, 2009 at 9:28 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668789#action_12668789 ]

    Sameer Paranjpye commented on HADOOP-4033:
    ------------------------------------------

    It should also be possible to set minspacestart to a reasonable value in mapred-site.xml, no?
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at Jan 30, 2009 at 9:40 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668791#action_12668791 ]

    Devaraj Das commented on HADOOP-4033:
    -------------------------------------

    Actually, I'd propose that we backport the 0.19 items - HADOOP-657 (and probably another related issue - HADOOP-9) to handle this issue.. HADOOP-657 seems like a better approach for the problem described.. thoughts?
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Devaraj Das (JIRA) at Jan 30, 2009 at 9:42 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668793#action_12668793 ]

    Devaraj Das commented on HADOOP-4033:
    -------------------------------------

    In addition to HADOOP-657, setting some sensible value for the minspacestart in mapred-site.xml should also be done
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Ravi Gummadi (JIRA) at Jan 30, 2009 at 9:52 am ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668795#action_12668795 ]

    Ravi Gummadi commented on HADOOP-4033:
    --------------------------------------

    Devaraj suggested porting JIRA-657's patch to branch 18 would be better. It applies fine to branch 18. Looking into the code of that patch to understand what is happening.
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Arun C Murthy (JIRA) at Jan 30, 2009 at 6:21 pm ⇧
    [ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668964#action_12668964 ]

    Arun C Murthy commented on HADOOP-4033:
    ---------------------------------------

    bq. Actually, I'd propose that we backport the 0.19 items - HADOOP-657 (and probably another related issue - HADOOP-9) to handle this issue.

    Uh, we can't add new features to a released branch?
    Task tracker should not ask for tasks if its temp disk space is almost full
    ---------------------------------------------------------------------------

    Key: HADOOP-4033
    URL: https://issues.apache.org/jira/browse/HADOOP-4033
    Project: Hadoop Core
    Issue Type: Bug
    Affects Versions: 0.17.1
    Reporter: Runping Qi
    Assignee: Ravi Gummadi

    I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
    Consequently, it had hard time to finish the tasks.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedAug 27, '08 at 1:15a
activeJan 30, '09 at 6:21p
posts9
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Arun C Murthy (JIRA): 9 posts

Content

People

Support

Translate

site design / logo © 2022 Grokbase