FAQ
Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
-----------------------------------------------------------------------------------------------------

Key: HADOOP-5994
URL: https://issues.apache.org/jira/browse/HADOOP-5994
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Devaraj Das
Assignee: Amar Kamat
Fix For: 0.21.0


Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Amar Kamat (JIRA) at Jun 10, 2009 at 9:20 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717998#action_12717998 ]

    Amar Kamat commented on HADOOP-5994:
    ------------------------------------

    Moving completed jobs to a done folder seems to be a big change to me. All the system build around hadoop will change. I propose to have running jobs in a _running_ folder and move the history file for completed jobs to the parent folder. This will involve lesser changes w.r.t external systems and testcases and also all the changes that will be needed will be during job submission/recovery only. Thoughts?
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 11, 2009 at 6:43 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718318#action_12718318 ]

    Amar Kamat commented on HADOOP-5994:
    ------------------------------------

    bq. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    I think for this we need to move completed files to a done folder.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 11, 2009 at 1:21 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5994:
    -------------------------------

    Attachment: HADOOP-5994-v1.7.patch

    Attaching a patch that moves completed jobs to a _done_ subfolder. Changes the testcases accordingly. Following are the changes
    # jobhistory now maintains an internal mapping from job-id to jobhistory-filename
    # finalize recovery reuses these cached values for renaming
    # jobtracker now invoked jobhistory.markAsCompleted() which moves the job to the done folder. the config files are also moved to the done folder
    # jobhistory's searching api's namely getJobHistoryFilename() and getJobHistoryLogLocation() now expect a boolean indicating if the job is runnning or complete
    # added a check in TestJobHistory to check if the filename returned by the framework after job completion belongs to a done folder or not, files are removed from the running folder. Same is also tested for config files too
    # HistoryCleaner now runs on done folder
    # Made sure that once the job files are moved to done folder, the framework (except historycleaner) is not affected by changes to done folder. History cleaner will fail and re-run upon an IOException.

    Running test-patch

    Tested this patch on a single node cluster and it works fine.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 12, 2009 at 3:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718704#action_12718704 ]

    Amar Kamat commented on HADOOP-5994:
    ------------------------------------

    Result of test-patch
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 6 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Running ant test now.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 12, 2009 at 7:32 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5994:
    -------------------------------

    Attachment: HADOOP-5994-v1.8.patch

    Fixed a small bug w.r.t. killing of un-initialized jobs. Result of test-patch
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 6 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Runnning ant-test now
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 12, 2009 at 12:00 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718810#action_12718810 ]

    Amar Kamat commented on HADOOP-5994:
    ------------------------------------

    Ant tests passed on my box.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jun 16, 2009 at 10:23 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720040#action_12720040 ]

    Hadoop QA commented on HADOOP-5994:
    -----------------------------------

    Tested these scenarios and found them to pass:

    1) Launch a job, allow it to complete and then verify if the related
    files move from history directory to done directory.

    2) Launch a job, kill it in the middle and verify if the related
    files move from history directory to done directory.

    3) Launch multiple jobs (say 5) simultaneously and kill them as soon
    as they start and verify if the related files move from history
    directory to done directory.

    4) Remove "done" directory and then kill the job to verify if done
    directory is created again.

    5) Kill a task before it says "setup successful".
    Expected behaviour: It moves the related files to done directory.
    Observed behaviour : same

    6) Older files under done folder can be removed without any issue to
    jobtracker.

    Expected behaviour: Files from "done" folder have to be removed after
    a specific time and job tracker should not be affected.

    Observed behaviour: Older files from "done" folder are removed after
    a successful job has been executed. job tracker si not affected by this.
    But the files don't get removed when jobs are killed. For this a Jira was raised:
    https://issues.apache.org/jira/browse/HADOOP-6050

    But the overall testscenario passes as the files in done directory is removed both by manually deleting it or by Jobtracker filecleaner deleting it, without affecting the running jobtracker.

    7) The webui of "Job History" works, whether the job is killed or it
    completes successfully. All the links in that "Job History" page is
    working.

    8) The in memory (running) jobs, shown in Jobtracker ui, works
    correctly. I checked all links.

    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Iyappan Srinivasan (JIRA) at Jun 16, 2009 at 10:25 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720042#action_12720042 ]

    Iyappan Srinivasan commented on HADOOP-5994:
    --------------------------------------------

    The previous test comment was put by me.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 18, 2009 at 10:45 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5994:
    -------------------------------

    Attachment: HADOOP-5994-v2.0.patch

    Attaching a new patch incorporating Devaraj's offline comments. Result of test-patch
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 6 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch, HADOOP-5994-v2.0.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Amar Kamat (JIRA) at Jun 19, 2009 at 7:41 am
    [ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Amar Kamat updated HADOOP-5994:
    -------------------------------

    Attachment: HADOOP-5994-v2.1.patch

    New patch with a bug fix, Result of test-patch
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 6 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Running ant test now.
    Move the completed jobs' history files to a DONE subdirectory inside the configured history directory
    -----------------------------------------------------------------------------------------------------

    Key: HADOOP-5994
    URL: https://issues.apache.org/jira/browse/HADOOP-5994
    Project: Hadoop Core
    Issue Type: Bug
    Components: mapred
    Reporter: Devaraj Das
    Assignee: Amar Kamat
    Fix For: 0.21.0

    Attachments: HADOOP-5994-v1.7.patch, HADOOP-5994-v1.8.patch, HADOOP-5994-v2.0.patch, HADOOP-5994-v2.1.patch


    Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedJun 9, '09 at 5:44a
activeJun 19, '09 at 7:41a
posts11
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Amar Kamat (JIRA): 11 posts

People

Translate

site design / logo © 2022 Grokbase