FAQ
Delete the jobconf copy from the log directory of the JobTracker when the job is retired
----------------------------------------------------------------------------------------

Key: HADOOP-5995
URL: https://issues.apache.org/jira/browse/HADOOP-5995
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Devaraj Das
Assignee: Amar Kamat
Fix For: 0.21.0


The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Amar Kamat (JIRA) at Jun 15, 2009 at 1:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5995:
    -------------------------------

    Attachment: HADOOP-5995-v1.0.patch

    Attaching a simple fix. Running test-patch now.
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 16, 2009 at 4:04 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719912#action_12719912 ]

    Amar Kamat commented on HADOOP-5995:
    ------------------------------------

    Result of test-patch
    [exec] -1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    The findbugs warning is to do with return value of file.delete(). I dont think its important. At the max we can log it. Running ant tests now.
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amareshwari Sriramadasu (JIRA) at Jun 16, 2009 at 6:39 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719946#action_12719946 ]

    Amareshwari Sriramadasu commented on HADOOP-5995:
    -------------------------------------------------

    Minor comments:
    1. change the comment from cleanup history to cleanup local conf
    2. Move the test from testJobHistoryUserLogLocation to testJobHistoryFile
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 16, 2009 at 7:47 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5995:
    -------------------------------

    Attachment: HADOOP-5995-v1.1.patch

    Attaching a patch incorporating Amareshwari's comments.
    Result of test-patch
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Running ant tests now
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amareshwari Sriramadasu (JIRA) at Jun 16, 2009 at 8:24 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719975#action_12719975 ]

    Amareshwari Sriramadasu commented on HADOOP-5995:
    -------------------------------------------------

    Changes look fine to me.
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Iyappan Srinivasan (JIRA) at Jun 17, 2009 at 8:24 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720557#action_12720557 ]

    Iyappan Srinivasan commented on HADOOP-5995:
    --------------------------------------------

    All test scenarios passed.

    Test scenarios are described:

    Parameters are set as:

    mapred.jobtracker.retirejob.check=10
    mapred.jobtracker.retirejob.interval=10

    1) A job is launched and killed. Conf.xml is removed from
    jobtracker/logdir after around 10 seconds.

    2) A job is launched and completed. Conf.xml is removed from
    jobtracker/logdir after around 10 seconds.

    3) After a job is completed and after around 10 seconds, the .xml file will dissapear from the logs directory and the job details should not be found from the "completed" section of jobtracker.jsp front page. But it should be found in the job Hisory.

    4) After a job is killed and after around 10 seconds, the .xml file will dissapear from the logs directory and the job details should not be found from the "Failed" section of jobtracker.jsp front page. But it should be found in the job Hisory.

    5) After launching multiple jobs of multiple types ( like sleep , randomwriter), allow them to complete successfully. After around 10 seconds, the corresponding .xml files will dissapear from the logs directory and the job details should not be found from the "completed" section of jobtracker.jsp front page. But they should be found in the job History.

    6) After launching multiple jobs of multiple types ( like sleep , randomwriter), and kill them. After around 10 seconds, the corresponding .xml files will dissapear from the logs directory and the job details should not be found from the "Failed" section of jobtracker.jsp front page. But they should be found in the job History.

    7) When the job has completed successsfully as well as when it is killed, the Job History should be showing properly will all links, inlucing job conf link showing proper values.

    8) Start a sleep job allow the map to complete 50% and then kill the job tracker. Restart the job tracker and find the job continuing from where it left off. Check the logs directory. Ther should be only one .xml file. After the job is complete, it will remove the .xml file after 10 seconds and that job will be seen in job tracker history link and not in the front page. Make sure all links are proper including the job conf.
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 17, 2009 at 9:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720584#action_12720584 ]

    Amar Kamat commented on HADOOP-5995:
    ------------------------------------

    Ant tests passed except
    Name||Type||Result||Link||
    org.apache.hadoop.mapred.TestJobTrackerRestartWithLostTracker|FAILED|Second time it passed|?|
    org.apache.hadoop.mapred.TestKillSubProcesses|FAILED|Passed second time|Failed on trunk too/HADOOP-6041|
    org.apache.hadoop.mapred.TestReduceFetch|FAILED|Known issue|HADOOP-6029|
    org.apache.hadoop.mapred.TestTaskTrackerMemoryManager|FAILED|Passed second time|?|
    test-contrib tests passed except
    Name||Type||Result||Link||
    org.apache.hadoop.streaming.TestStreamingExitStatus|FAILED|Known issue|HADOOP-5906
    org.apache.hadoop.streaming.TestStreamingStderr|FAILED (timeout)|Known issue|HADOOP-6062|
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 17, 2009 at 1:28 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720658#action_12720658 ]

    Amar Kamat commented on HADOOP-5995:
    ------------------------------------

    Found corresponding jira's for the test failures.
    HADOOP-6042 for TestJobTrackerRestartWithLostTracker
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 18, 2009 at 4:43 am
    [ https://issues.apache.org/jira/browse/HADOOP-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721056#action_12721056 ]

    Amar Kamat commented on HADOOP-5995:
    ------------------------------------

    Opened HADOOP-6075 for TestTaskTrackerMemoryManager failure.
    Delete the jobconf copy from the log directory of the JobTracker when the job is retired
    ----------------------------------------------------------------------------------------

    Key: HADOOP-5995
    URL: https://issues.apache.org/jira/browse/HADOOP-5995
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5995-v1.0.patch, HADOOP-5995-v1.1.patch


    The JobTracker (for web-ui viewing purposes), copies the jobconf from the hdfs and store it in the log directory. The file should be deleted when the job is retired (removed from memory).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 9, '09 at 5:54a
activeJun 18, '09 at 4:43a
posts10
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Amar Kamat (JIRA): 10 posts

People

Translate

site design / logo © 2022 Grokbase