FAQ
Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!


Michael

Search Discussions

  • Eli Collins at Mar 10, 2010 at 2:02 am
    Hey Michael,

    Symbolic links has been implemented [1] but are not yet available in a
    Hadoop release. The implementation is only available to clients that
    use the new FileContext API so clients like Hive need to be migrated
    from using FileSystem to FileContext. This is currently being done in
    Hadoop itself [2].

    Thanks,
    Eli

    [1] http://issues.apache.org/jira/browse/HDFS-245
    [2] http://issues.apache.org/jira/browse/HADOOP-6446

    On Tue, Mar 9, 2010 at 4:10 PM, jiang licht wrote:
    Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!


    Michael

  • Jiang licht at Mar 10, 2010 at 2:17 am
    Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand,  symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release?

    Thanks,

    Michael

    --- On Tue, 3/9/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Tuesday, March 9, 2010, 8:01 PM

    Hey Michael,

    Symbolic links has been implemented [1] but are not yet available in a
    Hadoop release. The implementation is only available to clients that
    use the new FileContext API so clients like Hive need to be migrated
    from using FileSystem to FileContext. This is currently being done in
    Hadoop itself [2].

    Thanks,
    Eli

    [1] http://issues.apache.org/jira/browse/HDFS-245
    [2] http://issues.apache.org/jira/browse/HADOOP-6446

    On Tue, Mar 9, 2010 at 4:10 PM, jiang licht wrote:
    Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!


    Michael

  • Eli Collins at Mar 10, 2010 at 5:09 pm
    Those patches won't apply to an existing release so you'd need to use trunk
    to get symlinks. They will be included in the next release, however only via
    the FileContext API.

    Thanks,
    Eli

    On Tue, Mar 9, 2010 at 6:16 PM, jiang licht wrote:

    Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
    I understand, symlink is available by using some relevant patches, right?
    From what you mentioned, it seems to me that symlink support is quite stable
    as of now. Just curious, why not included in a recent hadoop release yet or
    will it be considered in the next release?

    Thanks,

    Michael

    --- On Tue, 3/9/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Tuesday, March 9, 2010, 8:01 PM

    Hey Michael,

    Symbolic links has been implemented [1] but are not yet available in a
    Hadoop release. The implementation is only available to clients that
    use the new FileContext API so clients like Hive need to be migrated
    from using FileSystem to FileContext. This is currently being done in
    Hadoop itself [2].

    Thanks,
    Eli

    [1] http://issues.apache.org/jira/browse/HDFS-245
    [2] http://issues.apache.org/jira/browse/HADOOP-6446

    On Tue, Mar 9, 2010 at 4:10 PM, jiang licht wrote:
    Is there a way to create symlink in hdfs? And does LOAD function in Pig
    follows such a link? Thanks!

    Michael



  • Jiang licht at Mar 10, 2010 at 5:48 pm
    Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks.

    Thanks,

    Michael

    --- On Wed, 3/10/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Wednesday, March 10, 2010, 11:08 AM

    Those patches won't apply to an existing release so you'd need to use trunk
    to get symlinks. They will be included in the next release, however only via
    the FileContext API.

    Thanks,
    Eli

    On Tue, Mar 9, 2010 at 6:16 PM, jiang licht wrote:

    Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
    I understand,  symlink is available by using some relevant patches, right?
    From what you mentioned, it seems to me that symlink support is quite stable
    as of now. Just curious, why not included in a recent hadoop release yet or
    will it be considered in the next release?

    Thanks,

    Michael

    --- On Tue, 3/9/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Tuesday, March 9, 2010, 8:01 PM

    Hey Michael,

    Symbolic links has been implemented [1] but are not yet available in a
    Hadoop release. The implementation is only available to clients that
    use the new FileContext API so clients like Hive need to be migrated
    from using FileSystem to FileContext. This is currently being done in
    Hadoop itself [2].

    Thanks,
    Eli

    [1] http://issues.apache.org/jira/browse/HDFS-245
    [2] http://issues.apache.org/jira/browse/HADOOP-6446

    On Tue, Mar 9, 2010 at 4:10 PM, jiang licht wrote:
    Is there a way to create symlink in hdfs? And does LOAD function in Pig
    follows such a link? Thanks!

    Michael



  • Eli Collins at Mar 10, 2010 at 10:04 pm
    Yup, it needs to be ported to FileContext first (HADOOP-6424).

    Thanks,
    Eli
    On Wed, Mar 10, 2010 at 9:47 AM, jiang licht wrote:
    Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks.

    Thanks,

    Michael

    --- On Wed, 3/10/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Wednesday, March 10, 2010, 11:08 AM

    Those patches won't apply to an existing release so you'd need to use trunk
    to get symlinks. They will be included in the next release, however only via
    the FileContext API.

    Thanks,
    Eli

    On Tue, Mar 9, 2010 at 6:16 PM, jiang licht wrote:

    Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
    I understand,  symlink is available by using some relevant patches, right?
    From what you mentioned, it seems to me that symlink support is quite stable
    as of now. Just curious, why not included in a recent hadoop release yet or
    will it be considered in the next release?

    Thanks,

    Michael

    --- On Tue, 3/9/10, Eli Collins wrote:

    From: Eli Collins <eli@cloudera.com>
    Subject: Re: Symbolic link in Hadoop HDFS?
    To: common-user@hadoop.apache.org
    Date: Tuesday, March 9, 2010, 8:01 PM

    Hey Michael,

    Symbolic links has been implemented [1] but are not yet available in a
    Hadoop release. The implementation is only available to clients that
    use the new FileContext API so clients like Hive need to be migrated
    from using FileSystem to FileContext. This is currently being done in
    Hadoop itself [2].

    Thanks,
    Eli

    [1] http://issues.apache.org/jira/browse/HDFS-245
    [2] http://issues.apache.org/jira/browse/HADOOP-6446

    On Tue, Mar 9, 2010 at 4:10 PM, jiang licht wrote:
    Is there a way to create symlink in hdfs? And does LOAD function in Pig
    follows such a link? Thanks!

    Michael




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 10, '10 at 12:11a
activeMar 10, '10 at 10:04p
posts6
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Eli Collins: 3 posts Jiang licht: 3 posts

People

Translate

site design / logo © 2022 Grokbase