Grokbase › Groups › Hadoop › common-user › December 2008
FAQ
Hello all,

I normally upload files into hadoop via bin/hadoop fs -put file dest.

However, is there a way to somehow stream data into Hadoop?

For example, I'd love to do something like this:

zcat xxx >> HADOOP_HDFS_DESTINATION

This would save me a ton of time since I don't have to first unpack
the .tgz file and upload the raw file into HDFS.

Is this possible with Hadoop 0.19?

Thanks,
Ryan

Search Discussions

  • Alex Loddengaard at Dec 9, 2008 at 12:31 am ⇧
    This should answer your questions:

    <http://wiki.apache.org/hadoop/MountableHDFS>

    Alex
    On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte wrote:

    Hello all,

    I normally upload files into hadoop via bin/hadoop fs -put file dest.

    However, is there a way to somehow stream data into Hadoop?

    For example, I'd love to do something like this:

    zcat xxx >> HADOOP_HDFS_DESTINATION

    This would save me a ton of time since I don't have to first unpack
    the .tgz file and upload the raw file into HDFS.

    Is this possible with Hadoop 0.19?

    Thanks,
    Ryan
  • Ryan LeCompte at Dec 9, 2008 at 2:45 am ⇧
    Just what I need -- thanks!
    On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard wrote:
    This should answer your questions:

    <http://wiki.apache.org/hadoop/MountableHDFS>

    Alex
    On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte wrote:

    Hello all,

    I normally upload files into hadoop via bin/hadoop fs -put file dest.

    However, is there a way to somehow stream data into Hadoop?

    For example, I'd love to do something like this:

    zcat xxx >> HADOOP_HDFS_DESTINATION

    This would save me a ton of time since I don't have to first unpack
    the .tgz file and upload the raw file into HDFS.

    Is this possible with Hadoop 0.19?

    Thanks,
    Ryan
  • Aaron Kimball at Dec 10, 2008 at 3:36 am ⇧
    Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will use
    stdin.
    - Aaron
    On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte wrote:

    Just what I need -- thanks!
    On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard wrote:
    This should answer your questions:

    <http://wiki.apache.org/hadoop/MountableHDFS>

    Alex
    On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte wrote:

    Hello all,

    I normally upload files into hadoop via bin/hadoop fs -put file dest.

    However, is there a way to somehow stream data into Hadoop?

    For example, I'd love to do something like this:

    zcat xxx >> HADOOP_HDFS_DESTINATION

    This would save me a ton of time since I don't have to first unpack
    the .tgz file and upload the raw file into HDFS.

    Is this possible with Hadoop 0.19?

    Thanks,
    Ryan
  • Ryan LeCompte at Dec 10, 2008 at 4:19 am ⇧
    Even better! I'll try this out tomorrow.

    Thanks,
    Ryan



    On Dec 9, 2008, at 10:36 PM, "Aaron Kimball" wrote:

    Note also that "cat foo | bin/hadoop fs -put - some/hdfs/path" will
    use
    stdin.
    - Aaron
    On Mon, Dec 8, 2008 at 5:56 PM, Ryan LeCompte wrote:

    Just what I need -- thanks!

    On Mon, Dec 8, 2008 at 7:31 PM, Alex Loddengaard <[email protected]>
    wrote:
    This should answer your questions:

    <http://wiki.apache.org/hadoop/MountableHDFS>

    Alex

    On Mon, Dec 8, 2008 at 2:19 PM, Ryan LeCompte <[email protected]>
    wrote:
    Hello all,

    I normally upload files into hadoop via bin/hadoop fs -put file
    dest.

    However, is there a way to somehow stream data into Hadoop?

    For example, I'd love to do something like this:

    zcat xxx >> HADOOP_HDFS_DESTINATION

    This would save me a ton of time since I don't have to first unpack
    the .tgz file and upload the raw file into HDFS.

    Is this possible with Hadoop 0.19?

    Thanks,
    Ryan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedDec 8, '08 at 10:19p
activeDec 10, '08 at 4:19a
posts5
users3
websitehadoop.apache.org...
irc#hadoop

3 users in discussion

Ryan LeCompte: 3 posts Alex Loddengaard: 1 post Aaron Kimball: 1 post

Content

People

Support

Translate

site design / logo © 2023 Grokbase