[Hadoop-common-dev] [jira] Created: (HADOOP-2984) Distcp should have forrest documentation

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-2984:
------------------------------------

Priority: Blocker (was: Major)
Fix Version/s: 0.18.0
Assignee: Chris Douglas

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: 2984-0.patch

First draft

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: 2984-0.patch

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: (was: 2984-0.patch)

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603775#action_12603775 ]

Hadoop QA commented on HADOOP-2984:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383727/2984-0.patch
against trunk revision 665937.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

-1 release audit. The applied patch generated 203 release audit warnings (more than the trunk's current 201 warnings).

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2627/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2627/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2627/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2627/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2627/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: 2984-1.patch

Looks like I picked up an earlier draft without the license; trying again.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Status: Patch Available (was: Open)

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603970#action_12603970 ]

chris.douglas edited comment on HADOOP-2984 at 6/10/08 11:40 AM:
-----------------------------------------------------------------

I changed the name of the file, and might have missed the license somehow... trying again.

was (Author: chris.douglas):
Looks like I picked up an earlier draft without the license; trying again.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Status: Open (was: Patch Available)

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604016#action_12604016 ]

Hadoop QA commented on HADOOP-2984:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383764/2984-1.patch
against trunk revision 666056.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

-1 release audit. The applied patch generated 202 release audit warnings (more than the trunk's current 201 warnings).

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2632/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2632/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2632/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2632/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2632/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604367#action_12604367 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-2984:
------------------------------------------------

- Provide the full class name for DispCp once in the beginning for the document.

- Add a line to explain the command name distcp

- Some examples have long single line commands. These long single line commands got wrapped up in the docoument, especially in the pdf. It is good to use \ to break the long command into several lines. e.g.
Use
{noformat}
hadoop distcp \
hdfs://nn1:8020/foo/a \
hdfs://nn1:8020/foo/b \
hdfs://nn2:8020/bar/foo
{noformat}
instead of
{noformat}
hadoop distcp hdfs://nn1:8020/foo/a hdfs://nn1:8020/foo/b hdfs://nn2:8020/bar/foo
{noformat}

- I think it is more clear to add a command prompt for shell commands
e.g.
{noformat}
bash$ hadoop distcp ...
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604367#action_12604367 ]

szetszwo edited comment on HADOOP-2984 at 6/11/08 3:49 PM:
-------------------------------------------------------------------------

- Provide the full class name of DispCp in the beginning of the document.

- Add a line to explain the name distcp

- Some examples have long-single-line-commands. These long-single-line-commands got wrapped up in the document, especially in the pdf. It is good to use \ to break the long command into several lines. e.g.
Use
{noformat}
hadoop distcp \
hdfs://nn1:8020/foo/a \
hdfs://nn1:8020/foo/b \
hdfs://nn2:8020/bar/foo
{noformat}
instead of
{noformat}
hadoop distcp hdfs://nn1:8020/foo/a hdfs://nn1:8020/foo/b hdfs://nn2:8020/bar/foo
{noformat}

- I think it is more clear to add a command prompt for shell commands
e.g.
{noformat}
bash$ hadoop distcp ...
{noformat}


was (Author: szetszwo):
- Provide the full class name for DispCp once in the beginning for the document.

- Add a line to explain the command name distcp

- Some examples have long single line commands. These long single line commands got wrapped up in the docoument, especially in the pdf. It is good to use \ to break the long command into several lines. e.g.
Use
{noformat}
hadoop distcp \
hdfs://nn1:8020/foo/a \
hdfs://nn1:8020/foo/b \
hdfs://nn2:8020/bar/foo
{noformat}
instead of
{noformat}
hadoop distcp hdfs://nn1:8020/foo/a hdfs://nn1:8020/foo/b hdfs://nn2:8020/bar/foo
{noformat}

- I think it is more clear to add a command prompt for shell commands
e.g.
{noformat}
bash$ hadoop distcp ...
{noformat}

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604374#action_12604374 ]

Tsz Wo (Nicholas), SZE commented on HADOOP-2984:
------------------------------------------------

BTW, there are some unrelated changes of docs/hadoop-default.html in the patch.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: 2984-2.patch

bq. Provide the full class name of DispCp in the beginning of the document.

Since this is a guide to users of distcp and not developers, I left this out.

bq. Add a line to explain the name distcp
bq. Some examples have long-single-line-commands. These long-single-line-commands got wrapped up in the document, especially in the pdf. It is good to use \ to break the long command into several lines
bq. I think it is more clear to add a command prompt for shell commands

Good ideas; done.

Thanks for the review

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605001#action_12605001 ]

Koji Noguchi commented on HADOOP-2984:
--------------------------------------

+1.

bq. It's also worth noting that if another client is still writing to a source file, the copy will likely fail.

Maybe also mention,
* if any source files are deleted after distcp has started, mappers would fail (with file not found).
* If speculative execution is turned on as 'final', behavior of distcp is undefined.

bq. It's worth giving some examples of -update and -overwrite.

I always had trouble with these options.
Could you show how the target directory structures look like after the distcp?
(with and without -update/overwrite option)

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Attachment: 2984-3.patch

Incorporated Koji's feedback

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-2984:
----------------------------------

Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)

I just committed this.

[ https://issues.apache.org/jira/browse/HADOOP-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605432#action_12605432 ]

Hudson commented on HADOOP-2984:
--------------------------------

Integrated in Hadoop-trunk #520 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/520/])