FAQ
Hello my company is considering using a DFS for a project we're
currently working on. Since we don't have much experience in the field
I've compiled a list of questions that I hope can guide us to making
better decisions. I would greatly appreciate anyones help regarding
these issues.

- How do we handle failure of the single metaserver/namenode? Is there a
way to build a no "downtime" solution?

- What are the major differences between KFS and HDFS? - spec wise they
seem similar.

- Our service needs to handle a large amount of small (typically 1-20MB)
in size files. Is HDFS/KFS appropriate for this?

- Our service requires accessing these files in a low latency fashion,
we're less worried about throughput since (a) the files are small so
they might be cached by the OS and in any case will reside on a single
data server (probably won't have multiple chunks/blocks per file) and
(b) we don't have high throughput requirements. We do however require
low latency (lets say less than 50ms) to start getting data from the
file. Will HDFS/KFS provide those numbers?

- Are there any options for providing data reliability without the need
for complete replication (waisting storage space). For example
performing "raid xor" type operations between chunks/blocks?

- Are there any other DFS's you'd recommend looking into, which might
better fit our requirements?

Thanks,
Yoav.

Search Discussions

  • Yoav Steinberg at Oct 21, 2007 at 4:44 pm
    Hello my company is considering using a DFS for a project we're
    currently working on. Since we don't have much experience in the field
    I've compiled a list of questions that I hope can guide us to making
    better decisions. I would greatly appreciate anyones help regarding
    these issues.

    - How do we handle failure of the single metaserver/namenode? Is there a
    way to build a no "downtime" solution?

    - What are the major differences between KFS and HDFS? - spec wise they
    seem similar.

    - Our service needs to handle a large amount of small (typically 1-20MB)
    in size files. Is HDFS/KFS appropriate for this?

    - Our service requires accessing these files in a low latency fashion,
    we're less worried about throughput since (a) the files are small so
    they might be cached by the OS and in any case will reside on a single
    data server (probably won't have multiple chunks/blocks per file) and
    (b) we don't have high throughput requirements. We do however require
    low latency (lets say less than 50ms) to start getting data from the
    file. Will HDFS/KFS provide those numbers?

    - Are there any options for providing data reliability without the need
    for complete replication (waisting storage space). For example
    performing "raid xor" type operations between chunks/blocks?

    - Are there any other DFS's you'd recommend looking into, which might
    better fit our requirements?

    Thanks,
    Yoav.
  • Ted Dunning at Oct 21, 2007 at 5:27 pm

    On 10/21/07 4:13 AM, "Yoav Steinberg" wrote:

    - How do we handle failure of the single metaserver/namenode? Is there a
    way to build a no "downtime" solution?
    HDFS is currently designed in such a way that you should be able to achieve
    low downtime. It is not intended for zero downtime operation (yet).
    - What are the major differences between KFS and HDFS? - spec wise they
    seem similar.
    I don't know much about KFS, but from superficial examination, KFS is a bit
    more like a real file system (you can re-write files) but is written in C++
    by a separate team from the Hadoop developers. HDFS is written in Java and
    is maintained by the Hadoop team which leads to tighter tracking.

    I imagine that the KFS developers would know much more about this issue.
    - Our service needs to handle a large amount of small (typically 1-20MB)
    in size files. Is HDFS/KFS appropriate for this?
    Define "large amount". If you mean < 10-20 x 10^6, HDFS should be fine. If
    you mean 100 x 10^6, you will probably have problems.
    - Our service requires accessing these files in a low latency fashion,
    HDFS does this pretty well. You may, at some point, overload the namenode
    with file open traffic since it is involved in telling you where files.
    Hadoop is, as you have noted, optimized for reading large files. That means
    that the design assumption is that readers will spend much more time reading
    than opening. For your application this assumption is probably a bit wrong.
    - Are there any options for providing data reliability without the need
    for complete replication (waisting storage space). For example
    performing "raid xor" type operations between chunks/blocks?
    Replication is not as evil as you might think because it allows you to
    operate with very low cost storage devices and gives you increased read
    speeds while you are at it.

    You can easily achieve < $1 / GB of raw storage even at small volumes (I
    just bought a single rackmount Dell for home use that achieved this). For
    reasonably large installations (>8T), you can get near 0.5 $/GB without
    pushing matters that hard. This compares to several $/GB for any serious
    traditional storage solution such as NetApp, especially since you need to
    deal with replication for traditional solutions if you require 100% uptime.

    My practice grid at work has a dozen machines in it (more or less) that were
    cast-offs from normal production. Several of these machines "survived" a
    cooling system failure in the colocation facility and are distinctly flaky.
    Moreover, they are low priority machines subject to re-racking if higher
    priority tasks turn up. In several weeks of use, running with a replication
    level of 2, this batch of losers has not lost any data at all even though 3
    machines have failed, 2 have been taken down without notice to add disk and
    4 had to be relocated with little notice. This experience has really sold
    me on the superiority of distributed file systems based on simple
    replication.

    The net is the replication is a virtue, not really a problem.
    - Are there any other DFS's you'd recommend looking into, which might
    better fit our requirements?
    Look at MogileFS as well. It is designed for backing up large scale web
    properties. The namenode is kept in a database which makes read-only
    fallback operation relatively simple. It is also intended to support
    scaling to larger numbers of smaller objects than HDFS.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 21, '07 at 11:14a
activeOct 21, '07 at 5:27p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Yoav Steinberg: 2 posts Ted Dunning: 1 post

People

Translate

site design / logo © 2022 Grokbase