FAQ
distcp could have an option to preserve the full source path
------------------------------------------------------------

Key: HADOOP-5826
URL: https://issues.apache.org/jira/browse/HADOOP-5826
Project: Hadoop Core
Issue Type: Bug
Components: tools/distcp
Reporter: Rodrigo Schmidt
Assignee: Rodrigo Schmidt


It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • dhruba borthakur (JIRA) at May 18, 2009 at 5:06 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710273#action_12710273 ]

    dhruba borthakur commented on HADOOP-5826:
    ------------------------------------------

    distcp has options to preserve the modification time, replication factor, permissions, etc from the source file to the destination. One approach would be to introduce a command line parameter to distcp that "preserves" source-path.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 19, 2009 at 12:55 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710567#action_12710567 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Indeed, using the "-p" options sounds like a great idea.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 19, 2009 at 1:03 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710572#action_12710572 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------

    What does "preserving full source path" specifically mean?

    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 19, 2009 at 1:13 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710575#action_12710575 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Assuming option -p s did that, if we ran

    hadoop distcp -p s /a/b/c /d

    file/directory /d/a/b/c would be created instead of /d/c.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 19, 2009 at 7:05 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710645#action_12710645 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------


    That seems to be doable by user of distcp like :
    hadoop fs -mkdir dest:/d/a/b (copying properties of a and b from source, if possible)
    hadoop distcp src:/a/b/c dest:/d/a/b

    If we want to add this feature, I think it is better not to use '-p' for this. '-p' by convention implies preserving properties of each file.. it would be confusing if it also created these paths.


    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 19, 2009 at 5:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710802#action_12710802 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Another possibility is something like a basedir (-b) option that tells distcp what is the prefix of the source path that should be used as the base directory for copying.

    distcp -b /a /a/b/c /d

    would create file /d/b/c

    distcp -b / /a/b/c /d

    would creaet file /d/a/b/c

    this is more general than the original proposition.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 19, 2009 at 9:57 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710882#action_12710882 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    What should be the behavior in case basedir is not a correct prefix for one or more sources?

    I think it should throw an exception and cancel the copy.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 19, 2009 at 9:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710828#action_12710828 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    What about

    -basedir <dir>

    or

    -base <dir>

    ?
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 19, 2009 at 9:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710824#action_12710824 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------

    The basedir option sounds better. This would be a special option and I think it better to use a longer option name.

    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 19, 2009 at 10:39 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710839#action_12710839 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------

    -basedir with one or two examples in help should do I think.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt

    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 20, 2009 at 1:02 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: HADOOP-5826.patch
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 20, 2009 at 1:03 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Fix Version/s: 0.21.0
    Status: Patch Available (was: Open)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 21, 2009 at 9:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711829#action_12711829 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------


    I think it is better for someone with more familiarity with distcp to review.

    My comments from a brief look at the patch :

    - it might throw exception if used like 'distcp -basedir /a/b /a/b /dst'
    - does simple 'distcp /dir0/dir1 /dst' create /dst/dir1 directory or move contents for dir1 into /dst? If it is former, then test case does not seem to test feature.
    - The implementation changes the destination path but does not explicitly create the extra directories. Will it satisfy preserving the properties of those directories? (ie. '-basedir /a /a/b/c /dst' may not preserve properties for /dst/b and /dst/b/c).
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 21, 2009 at 10:27 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711844#action_12711844 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Thanks a lot for the comments!

    1) I'm throwing an exception on purpose when <basedir> equals some input source as I don't see why we should allow this.

    2) if /dst doesn't exist, /dst/dir1 is not created in the default behavior. As this is the case in the unit test, the feature is being tested (you can compare it to the unit test that copies files from dfs to dfs).

    3) You are definitely right about the extra directories' permissions. I didn't check anything about them and I should have. I'll change it and upload a new patch.

    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Raghu Angadi (JIRA) at May 21, 2009 at 11:07 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711852#action_12711852 ]

    Raghu Angadi commented on HADOOP-5826:
    --------------------------------------
    1) I'm throwing an exception on purpose when <basedir> equals some input source as I don't see why we should allow this.
    Even though it does not seem useful, it is logically correct way to use the option. Sometimes it might be required, say in automated scripts where argument to -basedir is not provided by human.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 22, 2009 at 12:08 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711873#action_12711873 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Ok! I think that will be automatically solved by my solution to point 3.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 22, 2009 at 6:48 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: HADOOP-5762.2.patch

    New patch!
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5762.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 22, 2009 at 9:41 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712276#action_12712276 ]

    Hadoop QA commented on HADOOP-5826:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12408779/HADOOP-5762.2.patch
    against trunk revision 777594.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/380/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/380/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/380/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/380/console

    This message is automatically generated.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5762.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at May 23, 2009 at 12:22 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712322#action_12712322 ]

    dhruba borthakur commented on HADOOP-5826:
    ------------------------------------------

    The test failures in streaming do not seem to be related to distcp at all.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5762.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 27, 2009 at 7:34 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: (was: HADOOP-5762.2.patch)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 27, 2009 at 8:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: HADOOP-5826.2.patch

    New patch
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 27, 2009 at 8:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: Patch Available (was: In Progress)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at May 27, 2009 at 8:01 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: In Progress (was: Patch Available)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at May 29, 2009 at 10:46 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714398#action_12714398 ]

    Hadoop QA commented on HADOOP-5826:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12409138/HADOOP-5826.2.patch
    against trunk revision 779807.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/423/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/423/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/423/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/423/console

    This message is automatically generated.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 1, 2009 at 10:12 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: Open (was: Patch Available)

    Minor bug in the previous patch
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 1, 2009 at 10:14 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: HADOOP-5826.3.patch
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 1, 2009 at 10:16 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: Patch Available (was: Open)

    New patch.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Jun 2, 2009 at 4:54 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715372#action_12715372 ]

    Hadoop QA commented on HADOOP-5826:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12409552/HADOOP-5826.3.patch
    against trunk revision 780875.

    +1 @author. The patch does not contain any @author tags.

    +1 tests included. The patch appears to include 3 new or modified tests.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 release audit. The applied patch does not increase the total number of release audit warnings.

    +1 core tests. The patch passed core unit tests.

    -1 contrib tests. The patch failed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/448/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/448/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/448/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/448/console

    This message is automatically generated.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 2, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715626#action_12715626 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Failed test seem to be completely unrelated to distcp.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 2, 2009 at 8:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715626#action_12715626 ]

    Rodrigo Schmidt edited comment on HADOOP-5826 at 6/2/09 1:36 PM:
    -----------------------------------------------------------------

    Failed test seems to be completely unrelated to distcp.

    was (Author: rschmidt):
    Failed test seem to be completely unrelated to distcp.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 9, 2009 at 8:20 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717603#action_12717603 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Anyone wants to review the latest patch?
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • dhruba borthakur (JIRA) at Jun 18, 2009 at 8:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721477#action_12721477 ]

    dhruba borthakur commented on HADOOP-5826:
    ------------------------------------------

    Code looks good. One comment:

    1. If basedir does not exist, then emit an error message saying "The directory specified by basedir does not exist".

    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 18, 2009 at 9:22 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721500#action_12721500 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    There are already two tests and two error messages that cover this case:

    - If basedir is not a prefix of the source (which will probably happen if it doesn't exist), it'll print: "Basedir [dir] is not a prefix of source path [src]"

    - If basedir is a prefix but is not a directory, it'll print: "Basedir [dir] is not a directory"

    Adding a new error message for the case where the directory doesn't exist looks like a redundant test given the first error case.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 18, 2009 at 11:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: Patch Available (was: Open)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.4.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 18, 2009 at 11:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Attachment: HADOOP-5826.4.patch

    Updated patch.
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.4.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 18, 2009 at 11:34 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Rodrigo Schmidt updated HADOOP-5826:
    ------------------------------------

    Status: Open (was: Patch Available)
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.4.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Rodrigo Schmidt (JIRA) at Jun 19, 2009 at 1:57 am
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721612#action_12721612 ]

    Rodrigo Schmidt commented on HADOOP-5826:
    -----------------------------------------

    Passed unit tests and ant test-patch returned the following:

    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    [exec]

    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Core
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.4.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Tom White (JIRA) at Jun 22, 2009 at 2:35 pm
    [ https://issues.apache.org/jira/browse/HADOOP-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Tom White updated HADOOP-5826:
    ------------------------------

    Status: Open (was: Patch Available)

    Unfortunately the patch no longer applies due to the project split. Would you be able to regenerate it for the MapReduce project please?
    distcp could have an option to preserve the full source path
    ------------------------------------------------------------

    Key: HADOOP-5826
    URL: https://issues.apache.org/jira/browse/HADOOP-5826
    Project: Hadoop Common
    Issue Type: Bug
    Components: tools/distcp
    Reporter: Rodrigo Schmidt
    Assignee: Rodrigo Schmidt
    Fix For: 0.21.0

    Attachments: HADOOP-5826.2.patch, HADOOP-5826.3.patch, HADOOP-5826.4.patch, HADOOP-5826.patch


    It would be helpful to have an option that preserves the full source path when copying files from one location to another. This is specially important when archiving/moving files from one cluster to another one.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 14, '09 at 1:17a
activeJun 22, '09 at 2:35p
posts39
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Tom White (JIRA): 39 posts

People

Translate

site design / logo © 2022 Grokbase