FAQ
This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
version) on a cluster of 2000 nodes. The programs we ran were RandomWriter
and Sort. Sort performance was pretty good - we could sort 20TB of data in
2.5 hours! Not many task failures - most of those that failed encountered
file checksum errors during merge and map output serving, some got killed
due to lack of progress reporting. Overall, a pretty successful run.

Search Discussions

  • Enis Soztutar at Sep 5, 2007 at 1:02 pm
    I am wondering how hadoop scores on sorting 1TB with say 1000 nodes. Is
    it possible for you to try the Terasort benchmark?


    Devaraj Das wrote:
    This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
    version) on a cluster of 2000 nodes. The programs we ran were RandomWriter
    and Sort. Sort performance was pretty good - we could sort 20TB of data in
    2.5 hours! Not many task failures - most of those that failed encountered
    file checksum errors during merge and map output serving, some got killed
    due to lack of progress reporting. Overall, a pretty successful run.

  • Devaraj Das at Sep 5, 2007 at 5:08 pm
    Will do so as soon as I get hold of a 1000 node cluster.
    -----Original Message-----
    From: Enis Soztutar
    Sent: Wednesday, September 05, 2007 6:32 PM
    To: hadoop-user@lucene.apache.org
    Subject: Re: Sort benchmark on 2000 nodes

    I am wondering how hadoop scores on sorting 1TB with say 1000
    nodes. Is it possible for you to try the Terasort benchmark?


    Devaraj Das wrote:
    This is FYI. We at Yahoo! could successfully run hadoop
    (upto date trunk
    version) on a cluster of 2000 nodes. The programs we ran
    were RandomWriter
    and Sort. Sort performance was pretty good - we could sort
    20TB of data in
    2.5 hours! Not many task failures - most of those that
    failed encountered
    file checksum errors during merge and map output serving,
    some got killed
    due to lack of progress reporting. Overall, a pretty successful run.

  • Joydeep Sen Sarma at Sep 5, 2007 at 5:20 pm
    It will be very useful to see the hadoop/job config settings and get
    some sense of the underlying hardware config.

    -----Original Message-----
    From: Devaraj Das
    Sent: Wednesday, September 05, 2007 2:29 AM
    To: hadoop-user@lucene.apache.org
    Subject: Sort benchmark on 2000 nodes

    This is FYI. We at Yahoo! could successfully run hadoop (upto date trunk
    version) on a cluster of 2000 nodes. The programs we ran were
    RandomWriter
    and Sort. Sort performance was pretty good - we could sort 20TB of data
    in
    2.5 hours! Not many task failures - most of those that failed
    encountered
    file checksum errors during merge and map output serving, some got
    killed
    due to lack of progress reporting. Overall, a pretty successful run.
  • Eric Baldeschwieler at Sep 6, 2007 at 8:08 am
    hardware is similar top that discussed here:

    http://wiki.apache.org/lucene-hadoop-data/attachments/
    HadoopPresentations/attachments/oscon-part-2.pdf

    - 10:1 oversubscribed network (so 100mBit bandwidth all nodes to all
    nodes)
    - 40 nodes / leaf switch
    - Machines are beefy
    - 4SATA drives, 500 or 750 GB each, 7200 RPM
    - 4+ cores (modern Intels or AMDs)
    - 4+ GB RAM
    On Sep 5, 2007, at 10:19 AM, Joydeep Sen Sarma wrote:

    It will be very useful to see the hadoop/job config settings and get
    some sense of the underlying hardware config.

    -----Original Message-----
    From: Devaraj Das
    Sent: Wednesday, September 05, 2007 2:29 AM
    To: hadoop-user@lucene.apache.org
    Subject: Sort benchmark on 2000 nodes

    This is FYI. We at Yahoo! could successfully run hadoop (upto date
    trunk
    version) on a cluster of 2000 nodes. The programs we ran were
    RandomWriter
    and Sort. Sort performance was pretty good - we could sort 20TB of
    data
    in
    2.5 hours! Not many task failures - most of those that failed
    encountered
    file checksum errors during merge and map output serving, some got
    killed
    due to lack of progress reporting. Overall, a pretty successful run.
  • Devaraj Das at Sep 6, 2007 at 12:54 pm
    Thanks Eric for pointing out the hardware spec. I have updated the
    hadoop-config on the hadoop FAQ - http://wiki.apache.org/lucene-hadoop/FAQ
    -----Original Message-----
    From: Eric Baldeschwieler
    Sent: Thursday, September 06, 2007 1:37 PM
    To: hadoop-user@lucene.apache.org
    Subject: Re: Sort benchmark on 2000 nodes

    hardware is similar top that discussed here:

    http://wiki.apache.org/lucene-hadoop-data/attachments/
    HadoopPresentations/attachments/oscon-part-2.pdf

    - 10:1 oversubscribed network (so 100mBit bandwidth all nodes to all
    nodes)
    - 40 nodes / leaf switch
    - Machines are beefy
    - 4SATA drives, 500 or 750 GB each, 7200 RPM
    - 4+ cores (modern Intels or AMDs)
    - 4+ GB RAM
    On Sep 5, 2007, at 10:19 AM, Joydeep Sen Sarma wrote:

    It will be very useful to see the hadoop/job config
    settings and get
    some sense of the underlying hardware config.

    -----Original Message-----
    From: Devaraj Das
    Sent: Wednesday, September 05, 2007 2:29 AM
    To: hadoop-user@lucene.apache.org
    Subject: Sort benchmark on 2000 nodes

    This is FYI. We at Yahoo! could successfully run hadoop (upto date
    trunk
    version) on a cluster of 2000 nodes. The programs we ran were
    RandomWriter and Sort. Sort performance was pretty good - we could
    sort 20TB of data in
    2.5 hours! Not many task failures - most of those that failed
    encountered file checksum errors during merge and map
    output serving,
    some got killed due to lack of progress reporting. Overall, a pretty
    successful run.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedSep 5, '07 at 9:30a
activeSep 6, '07 at 12:54p
posts6
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase