FAQ
Hi,

I'd like to move and copy files from one directory in HDFS to another
one. I know there are methods in the Filesystem API that enable
copying files between the local disk and HDFS, but I couldn't figure
out how to do this between two paths both in HDFS. I think rename(Path
src, Path dest) can be used to move files, but copying still remains a
challenge to me. Any ideas?

Thanks,
Jim

Search Discussions

  • Lohit at May 13, 2011 at 9:01 pm
    There is no FileSystem API to copy.
    You could try
    hadoop dfs -cp <src> <dest>

    which basically reads the file and writes to new file.
    The code for this is in FsShell.java


    ----- Original Message ----
    From: Jim Twensky <jim.twensky@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Fri, May 13, 2011 1:21:09 PM
    Subject: FileSystem API - Moving files in HDFS

    Hi,

    I'd like to move and copy files from one directory in HDFS to another
    one. I know there are methods in the Filesystem API that enable
    copying files between the local disk and HDFS, but I couldn't figure
    out how to do this between two paths both in HDFS. I think rename(Path
    src, Path dest) can be used to move files, but copying still remains a
    challenge to me. Any ideas?

    Thanks,
    Jim
  • Mahadev Konar at May 14, 2011 at 5:49 am
    Jim,
    you can use FileUtil.copy() methods to copy files.

    Hope that helps.


    --
    thanks
    mahadev
    @mahadevkonar


    On Fri, May 13, 2011 at 2:00 PM, lohit wrote:
    There is no FileSystem API to copy.
    You could try
    hadoop dfs -cp <src> <dest>

    which basically reads the file and writes to new file.
    The code for this is in FsShell.java


    ----- Original Message ----
    From: Jim Twensky <jim.twensky@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Fri, May 13, 2011 1:21:09 PM
    Subject: FileSystem API - Moving files in HDFS

    Hi,

    I'd like to move and copy files from one directory in HDFS to another
    one. I know there are methods in the Filesystem API that enable
    copying files between the local disk and HDFS, but I couldn't figure
    out how to do this between two paths both in HDFS. I think rename(Path
    src, Path dest) can be used to move files, but copying still remains a
    challenge to me. Any ideas?

    Thanks,
    Jim
  • Robert Evans at May 16, 2011 at 5:08 pm
    If they are lots of large files, and you need to copy them quickly, i.e. Not have all the data go through a single machine, you can use hadoop distcp too.

    --Bobby

    On 5/14/11 12:49 AM, "Mahadev Konar" wrote:

    Jim,
    you can use FileUtil.copy() methods to copy files.

    Hope that helps.


    --
    thanks
    mahadev
    @mahadevkonar


    On Fri, May 13, 2011 at 2:00 PM, lohit wrote:
    There is no FileSystem API to copy.
    You could try
    hadoop dfs -cp <src> <dest>

    which basically reads the file and writes to new file.
    The code for this is in FsShell.java


    ----- Original Message ----
    From: Jim Twensky <jim.twensky@gmail.com>
    To: core-user@hadoop.apache.org
    Sent: Fri, May 13, 2011 1:21:09 PM
    Subject: FileSystem API - Moving files in HDFS

    Hi,

    I'd like to move and copy files from one directory in HDFS to another
    one. I know there are methods in the Filesystem API that enable
    copying files between the local disk and HDFS, but I couldn't figure
    out how to do this between two paths both in HDFS. I think rename(Path
    src, Path dest) can be used to move files, but copying still remains a
    challenge to me. Any ideas?

    Thanks,
    Jim

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 13, '11 at 8:21p
activeMay 16, '11 at 5:08p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase