[Hadoop-common-dev] [jira] Created: (HADOOP-5994) Move the completed jobs' history files to a DONE subdirectory inside the configured history directory

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717998#action_12717998 ]

Amar Kamat commented on HADOOP-5994:
------------------------------------

Moving completed jobs to a done folder seems to be a big change to me. All the system build around hadoop will change. I propose to have running jobs in a _running_ folder and move the history file for completed jobs to the parent folder. This will involve lesser changes w.r.t external systems and testcases and also all the changes that will be needed will be during job submission/recovery only. Thoughts?

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718318#action_12718318 ]

Amar Kamat commented on HADOOP-5994:
------------------------------------

bq. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).
I think for this we need to move completed files to a done folder.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5994:
-------------------------------

Attachment: HADOOP-5994-v1.7.patch

Attaching a patch that moves completed jobs to a _done_ subfolder. Changes the testcases accordingly. Following are the changes
# jobhistory now maintains an internal mapping from job-id to jobhistory-filename
# finalize recovery reuses these cached values for renaming
# jobtracker now invoked jobhistory.markAsCompleted() which moves the job to the done folder. the config files are also moved to the done folder
# jobhistory's searching api's namely getJobHistoryFilename() and getJobHistoryLogLocation() now expect a boolean indicating if the job is runnning or complete
# added a check in TestJobHistory to check if the filename returned by the framework after job completion belongs to a done folder or not, files are removed from the running folder. Same is also tested for config files too
# HistoryCleaner now runs on done folder
# Made sure that once the job files are moved to done folder, the framework (except historycleaner) is not affected by changes to done folder. History cleaner will fail and re-run upon an IOException.

Running test-patch

Tested this patch on a single node cluster and it works fine.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718704#action_12718704 ]

Amar Kamat commented on HADOOP-5994:
------------------------------------

Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

Running ant test now.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5994:
-------------------------------

Attachment: HADOOP-5994-v1.8.patch

Fixed a small bug w.r.t. killing of un-initialized jobs. Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

Runnning ant-test now

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718810#action_12718810 ]

Amar Kamat commented on HADOOP-5994:
------------------------------------

Ant tests passed on my box.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720040#action_12720040 ]

Hadoop QA commented on HADOOP-5994:
-----------------------------------

Tested these scenarios and found them to pass:

1) Launch a job, allow it to complete and then verify if the related
files move from history directory to done directory.

2) Launch a job, kill it in the middle and verify if the related
files move from history directory to done directory.

3) Launch multiple jobs (say 5) simultaneously and kill them as soon
as they start and verify if the related files move from history
directory to done directory.

4) Remove "done" directory and then kill the job to verify if done
directory is created again.

5) Kill a task before it says "setup successful".
Expected behaviour: It moves the related files to done directory.
Observed behaviour : same

6) Older files under done folder can be removed without any issue to
jobtracker.

Expected behaviour: Files from "done" folder have to be removed after
a specific time and job tracker should not be affected.

Observed behaviour: Older files from "done" folder are removed after
a successful job has been executed. job tracker si not affected by this.
But the files don't get removed when jobs are killed. For this a Jira was raised:
https://issues.apache.org/jira/browse/HADOOP-6050

But the overall testscenario passes as the files in done directory is removed both by manually deleting it or by Jobtracker filecleaner deleting it, without affecting the running jobtracker.

7) The webui of "Job History" works, whether the job is killed or it
completes successfully. All the links in that "Job History" page is
working.

8) The in memory (running) jobs, shown in Jobtracker ui, works
correctly. I checked all links.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720042#action_12720042 ]

Iyappan Srinivasan commented on HADOOP-5994:
--------------------------------------------

The previous test comment was put by me.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5994:
-------------------------------

Attachment: HADOOP-5994-v2.0.patch

Attaching a new patch incorporating Devaraj's offline comments. Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

[ https://issues.apache.org/jira/browse/HADOOP-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-5994:
-------------------------------

Attachment: HADOOP-5994-v2.1.patch

New patch with a bug fix, Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

Running ant test now.