FAQ
So, I use hdfs to store very large files and access them thru various client
(100 clients) using FS utils. Are there any other tools or projects
that solely use hdfs as its storage for fast access? I know hbase uses it
but requires mapreduce. I want to know only hdfs without mapreduce.



--
--- Get your facts first, then you can distort them as you please.--

Search Discussions

  • Joey Echeverria at Jul 16, 2011 at 3:52 pm
    HBase does not require MapReduce.

    -Joey


    On Jul 16, 2011, at 11:46, Rita wrote:

    So, I use hdfs to store very large files and access them thru various client (100 clients) using FS utils. Are there any other tools or projects that solely use hdfs as its storage for fast access? I know hbase uses it but requires mapreduce. I want to know only hdfs without mapreduce.



    --
    --- Get your facts first, then you can distort them as you please.--
  • Owen O'Malley at Jul 16, 2011 at 4:15 pm
    The scientists at CERN use HDFS for storing their large data sets and
    don't use MapReduce at all. (I believe most of their applications are
    MPI.) At Yahoo, some users run non-MapReduce in the shared MapReduce
    cluster.

    -- Owen
    On Jul 16, 2011, at 8:46 AM, Rita wrote:

    So, I use hdfs to store very large files and access them thru various client (100 clients) using FS utils. Are there any other tools or projects that solely use hdfs as its storage for fast access? I know hbase uses it but requires mapreduce. I want to know only hdfs without mapreduce.



    --
    --- Get your facts first, then you can distort them as you please.--
  • Dhruba Borthakur at Jul 16, 2011 at 5:40 pm
    We are using hdfs for backups (and archival) of a huge number of databases

    Thanks
    Dhruba

    Sent from my iPhone
    On Jul 16, 2011, at 9:14 AM, "Owen O'Malley" wrote:

    The scientists at CERN use HDFS for storing their large data sets and
    don't use MapReduce at all. (I believe most of their applications are
    MPI.) At Yahoo, some users run non-MapReduce in the shared MapReduce
    cluster.

    -- Owen
    On Jul 16, 2011, at 8:46 AM, Rita wrote:

    So, I use hdfs to store very large files and access them thru various client (100 clients) using FS utils. Are there any other tools or projects that solely use hdfs as its storage for fast access? I know hbase uses it but requires mapreduce. I want to know only hdfs without mapreduce.



    --
    --- Get your facts first, then you can distort them as you please.--
  • Will Maier at Jul 17, 2011 at 5:47 am
    Hi Owen-
    On Sat, Jul 16, 2011 at 09:14:59AM -0700, Owen O'Malley wrote:
    The scientists at CERN use HDFS for storing their large data sets and
    don't use MapReduce at all. (I believe most of their applications are
    MPI.) At Yahoo, some users run non-MapReduce in the shared MapReduce
    cluster.
    Various groups supporting LHC physics at CERN do have about a dozen
    petabyte-sized HDFS clusters. Our workflows are highly parallel, so we mostly
    use traditional schedulers like Condor[0], PBS and LSF.

    Thanks!

    [0] http://www.cs.wisc.edu/condor

    --

    Will Maier - UW High Energy Physics
    cel: 608.438.6162
    tel: 608.263.9692
    web: http://www.hep.wisc.edu/~wcmaier/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedJul 16, '11 at 3:46p
activeJul 17, '11 at 5:47a
posts5
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase