FAQ
So my company is looking at only using dell or hp for our
hadoop cluster and a sun thumper to backup the data. The prices are
ok, after a 40% discount, but realistically I am paying twice as much
as if I went to silicon mechanics, and with a much slower machine. It
seems as though the big expense are the disks. Even with a 40%
discount 550$ per 1tb disk seems crazy expensive. Also, they are
pushing me to build a smaller cluster (6 nodes) and I am pushing back
for nodes half the size but having twice as many. So how much of a
performance difference can I expect btwn 12 nodes with 1 xeon 5 series
running at 2.26 ghz 8 gigs of ram with 4 1 tb disks and a 6 node
cluster with 2 xeon 5 series running at 2.26 16 gigs of ram with 8 1
tb disks. Both setups will also have 2 very small sata drives in raid
1 for the OS. I will be doing some stuff with hadoop and a lot of
stuff with HBase. What are the considerations with HDFS performance
with a low number of nodes,etc.

Search Discussions

  • Brian Bockelman at Oct 15, 2009 at 4:06 pm
    Hey Alex,

    In order to lower cost, you'll probably want to order the worker nodes
    without hard drives then buy them separately. HDFS provides a
    software-level RAID, so most of the reasonings behind buying hard
    drives from Dell/HP are irrelevant - you are just paying an extra $400
    per hard drive. I know Dell sells the R410 which has 4 SATA bays; I'm
    sure Steve knows an HP model that has something similar.

    However, BE VERY CAREFUL when you do this. From experience, a certain
    large manufacturer (I don't know about Dell/HP) will refuse to ship
    (or sell separately) hard drive trays if you order their machine
    without hard drives. When this happened to us, we were not able to
    return the machines because they were custom orders. Eventually, we
    had to get someone to go to the machine shop and build 72 hard drive
    trays for us.

    Worst. Experience. Ever.

    So, ALWAYS ASK and make sure that you can buy empty hard drive trays
    for that specific model (or at least that it ships with them).

    Brian
    On Oct 15, 2009, at 10:48 AM, Alex Newman wrote:

    So my company is looking at only using dell or hp for our
    hadoop cluster and a sun thumper to backup the data. The prices are
    ok, after a 40% discount, but realistically I am paying twice as much
    as if I went to silicon mechanics, and with a much slower machine. It
    seems as though the big expense are the disks. Even with a 40%
    discount 550$ per 1tb disk seems crazy expensive. Also, they are
    pushing me to build a smaller cluster (6 nodes) and I am pushing back
    for nodes half the size but having twice as many. So how much of a
    performance difference can I expect btwn 12 nodes with 1 xeon 5 series
    running at 2.26 ghz 8 gigs of ram with 4 1 tb disks and a 6 node
    cluster with 2 xeon 5 series running at 2.26 16 gigs of ram with 8 1
    tb disks. Both setups will also have 2 very small sata drives in raid
    1 for the OS. I will be doing some stuff with hadoop and a lot of
    stuff with HBase. What are the considerations with HDFS performance
    with a low number of nodes,etc.
  • Steve Loughran at Oct 15, 2009 at 4:43 pm

    Brian Bockelman wrote:
    Hey Alex,

    In order to lower cost, you'll probably want to order the worker nodes
    without hard drives then buy them separately. HDFS provides a
    software-level RAID, so most of the reasonings behind buying hard drives
    from Dell/HP are irrelevant - you are just paying an extra $400 per hard
    drive. I know Dell sells the R410 which has 4 SATA bays; I'm sure Steve
    knows an HP model that has something similar.
    I will start the official disclaimer "I make no recommendations about
    hardware" here, so as not to get into trouble. Talk to you reseller or
    account team.

    You can get servers with lots of drives in them DL180 and SL170z are
    acronyms that spring to mind.
    Big issues to consider
    * server:CPU ratio
    * power budget
    * rack weight
    * do you ever plan to stick in more CPUs? Some systems take this,
    others don't.
    * Intel vs AMD
    * How much ECC RAM can you afford. And yes, it must be ECC.

    server disks are higher RPM and specced for more hours than "consumer"
    disks, I don't know what that means in terms of lifespan, but the RPM
    translates into bandwidth off the disk.
    However, BE VERY CAREFUL when you do this. From experience, a certain
    large manufacturer (I don't know about Dell/HP) will refuse to ship (or
    sell separately) hard drive trays if you order their machine without
    hard drives. When this happened to us, we were not able to return the
    machines because they were custom orders. Eventually, we had to get
    someone to go to the machine shop and build 72 hard drive trays for us.
    that is what physics PhD students are for, at least they didn't get a
    lifetimes radiation dose for this job
    Worst. Experience. Ever.

    So, ALWAYS ASK and make sure that you can buy empty hard drive trays for
    that specific model (or at least that it ships with them).

    Brian
    On Oct 15, 2009, at 10:48 AM, Alex Newman wrote:

    So my company is looking at only using dell or hp for our
    hadoop cluster and a sun thumper to backup the data. The prices are
    ok, after a 40% discount, but realistically I am paying twice as much
    as if I went to silicon mechanics, and with a much slower machine. It
    seems as though the big expense are the disks. Even with a 40%
    discount 550$ per 1tb disk seems crazy expensive. Also, they are
    pushing me to build a smaller cluster (6 nodes) and I am pushing back
    for nodes half the size but having twice as many. So how much of a
    performance difference can I expect btwn 12 nodes with 1 xeon 5 series
    running at 2.26 ghz 8 gigs of ram with 4 1 tb disks and a 6 node
    cluster with 2 xeon 5 series running at 2.26 16 gigs of ram with 8 1
    tb disks. Both setups will also have 2 very small sata drives in raid
    1 for the OS. I will be doing some stuff with hadoop and a lot of
    stuff with HBase. What are the considerations with HDFS performance
    with a low number of nodes,etc.

    It's an interesting Q as to what is better, fewer nodes with more
    storage/CPU or more, smaller nodes.

    Bigger servers
    * more chance of running code near the data
    * less data moved over the LAN at shuffle time
    * RAM consumption can be more agile across tasks.
    * increased chance of disk failure on a node; hadoop handles that very
    badly right now (pre 0.20 -datanode goes offline)

    Smaller servers
    * easier to place data redundantly across machines
    * less RAM taken up by other people's jobs
    * more nodes stay up when a disk fails (less important on 0.20 onwards)
    * when a node goes down, less data to re-replicate across the other
    machines

    1. I would like to hear other people's opinions,

    2. The gridmix 2 benchmarking stuff tries to create synthetic benchmarks
    from your real data runs. Try that, collect some data, then go to your
    suppliers.

    -Steve

    COI disclaimer signature:
    -----------------------
    Hewlett-Packard Limited
    Registered Office: Cain Road, Bracknell, Berks RG12 1HN
    Registered No: 690597 England
  • Allen Wittenauer at Oct 15, 2009 at 5:54 pm

    On 10/15/09 9:42 AM, "Steve Loughran" wrote:
    It's an interesting Q as to what is better, fewer nodes with more
    storage/CPU or more, smaller nodes.

    Bigger servers
    * more chance of running code near the data
    * less data moved over the LAN at shuffle time
    * RAM consumption can be more agile across tasks.
    * increased chance of disk failure on a node; hadoop handles that very
    badly right now (pre 0.20 -datanode goes offline)

    Smaller servers
    * easier to place data redundantly across machines
    * less RAM taken up by other people's jobs
    * more nodes stay up when a disk fails (less important on 0.20 onwards)
    * when a node goes down, less data to re-replicate across the other
    machines

    1. I would like to hear other people's opinions,
    - Don't forget the about the more obvious things: if you go with more disks
    per server, that also means likely means less controllers doing IO.

    - Keep in mind that fewer CPUs/less RAM=less task slots available. While
    your workflow may not be CPU-bound in the traditional sense, if you are
    spawning 5000 maps, you're going to need quite a few slots to get your work
    done in a reasonable time.

    - To counter that, it seems we can run more tasks-per-node in LI's 2U config
    than Y!'s 1U config. But this might be an apples/oranges comparison (LI
    uses Solaris+ZFS, Y! uses Linux+ext3).
    2. The gridmix 2 benchmarking stuff tries to create synthetic benchmarks
    from your real data runs. Try that, collect some data, then go to your
    suppliers.
    +1
  • Tim robertson at Oct 15, 2009 at 7:21 pm
    Hi Alex,

    I am also doing a little on HBase - I think I have heard that a few
    higher memory machines with more spindles and cores per machine beat
    more smaller machines with similar total capacity (I *guess* this is
    due to memory buffers and data locality). Please don't take my word
    for it, but I recommend posting the same to the HBase list -
    http://hadoop.apache.org/hbase/mailing_lists.html#Users.

    Cheers,
    Tim



    On Thu, Oct 15, 2009 at 7:54 PM, Allen Wittenauer
    wrote:
    On 10/15/09 9:42 AM, "Steve Loughran" wrote:
    It's an interesting Q as to what is better, fewer nodes with more
    storage/CPU or more, smaller nodes.

    Bigger servers
    * more chance of running code near the data
    * less data moved over the LAN at shuffle time
    * RAM consumption can be more agile across tasks.
    * increased chance of disk failure on a node; hadoop handles that very
    badly right now (pre 0.20 -datanode goes offline)

    Smaller servers
    * easier to place data redundantly across machines
    * less RAM taken up by other people's jobs
    * more nodes stay up when a disk fails (less important on 0.20 onwards)
    * when a node goes down, less data to re-replicate across the other
    machines

    1. I would like to hear other people's opinions,
    - Don't forget the about the more obvious things:  if you go with more disks
    per server, that also means likely means less controllers doing IO.

    - Keep in mind that fewer CPUs/less RAM=less task slots available.  While
    your workflow may not be CPU-bound in the traditional sense, if you are
    spawning 5000 maps, you're going to need quite a few slots to get your work
    done in a reasonable time.

    - To counter that, it seems we can run more tasks-per-node in LI's 2U config
    than Y!'s 1U config.  But this might be an apples/oranges comparison (LI
    uses Solaris+ZFS, Y! uses Linux+ext3).
    2. The gridmix 2 benchmarking stuff tries to create synthetic benchmarks
    from your real data runs. Try that, collect some data, then go to your
    suppliers.
    +1
  • Patrick Angeles at Oct 15, 2009 at 4:12 pm
    After the discount, an equivalently configured Dell comes about 10-20% over
    the Silicon Mechanics price. It's close enough that unless you're spending
    100k it won't make that much of a difference. Talk to a rep, call them out
    on the ridiculous drive pricing, buy at the end of their fiscal quarter.
    Strip down the machines (no RAID cards, no DVD/CD drive, non-redundant power
    supply, etc.) to get the price lower. No need for dedicated SATA drives with
    RAID for your OS. Most of that is accessed during boot time so it won't
    contend that much with HDFS.

    We just got a bunch of Dell R410s with 24GB ram, 2x2.26Ghz procs and 4x1TB
    drives.

    I would go for beefier nodes with less quantity. Of course, some of this
    depends on the volume of data and type of processing that you do. If you're
    running HBase, you would benefit from lots of RAM. You also have to remember
    that dual socket configs are more power efficient so you can fit more in a
    single rack.

    Cheers,

    - P
    On Thu, Oct 15, 2009 at 11:48 AM, Alex Newman wrote:

    So my company is looking at only using dell or hp for our
    hadoop cluster and a sun thumper to backup the data. The prices are
    ok, after a 40% discount, but realistically I am paying twice as much
    as if I went to silicon mechanics, and with a much slower machine. It
    seems as though the big expense are the disks. Even with a 40%
    discount 550$ per 1tb disk seems crazy expensive. Also, they are
    pushing me to build a smaller cluster (6 nodes) and I am pushing back
    for nodes half the size but having twice as many. So how much of a
    performance difference can I expect btwn 12 nodes with 1 xeon 5 series
    running at 2.26 ghz 8 gigs of ram with 4 1 tb disks and a 6 node
    cluster with 2 xeon 5 series running at 2.26 16 gigs of ram with 8 1
    tb disks. Both setups will also have 2 very small sata drives in raid
    1 for the OS. I will be doing some stuff with hadoop and a lot of
    stuff with HBase. What are the considerations with HDFS performance
    with a low number of nodes,etc.
  • Edward Capriolo at Oct 15, 2009 at 4:33 pm

    On Thu, Oct 15, 2009 at 12:12 PM, Patrick Angeles wrote:
    After the discount, an equivalently configured Dell comes about 10-20% over
    the Silicon Mechanics price. It's close enough that unless you're spending
    100k it won't make that much of a difference. Talk to a rep, call them out
    on the ridiculous drive pricing, buy at the end of their fiscal quarter.
    Strip down the machines (no RAID cards, no DVD/CD drive, non-redundant power
    supply, etc.) to get the price lower. No need for dedicated SATA drives with
    RAID for your OS. Most of that is accessed during boot time so it won't
    contend that much with HDFS.

    We just got a bunch of Dell R410s with 24GB ram, 2x2.26Ghz procs and 4x1TB
    drives.

    I would go for beefier nodes with less quantity.  Of course, some of this
    depends on the volume of data and type of processing that you do. If you're
    running HBase, you would benefit from lots of RAM. You also have to remember
    that dual socket configs are more power efficient so you can fit more in a
    single rack.

    Cheers,

    - P
    On Thu, Oct 15, 2009 at 11:48 AM, Alex Newman wrote:

    So my company is looking at only using dell or hp for our
    hadoop cluster and a sun thumper to backup the data. The prices are
    ok, after a 40% discount, but realistically I am paying twice as much
    as if I went to silicon mechanics, and with a much slower machine. It
    seems as though the big expense are the disks. Even with a 40%
    discount 550$ per 1tb disk seems crazy expensive. Also, they are
    pushing me to build a smaller cluster (6 nodes) and I am pushing back
    for nodes half the size but having twice as many. So how much of a
    performance difference can I expect btwn 12 nodes with 1 xeon 5 series
    running at 2.26 ghz 8 gigs of ram with 4 1 tb disks and a 6 node
    cluster with 2 xeon 5 series running at 2.26 16 gigs of ram with 8 1
    tb disks. Both setups will also have 2 very small sata drives in raid
    1 for the OS. I will be doing some stuff with hadoop and a lot of
    stuff with HBase. What are the considerations with HDFS performance
    with a low number of nodes,etc.
    No need for dedicated SATA drives with
    RAID for your OS. Most of that is accessed during boot time so it won't
    contend that much with HDFS.
    You may want to RAID your OS. If you lose a datanode with a large
    volume of data say (8 TB) Hadoop will begin the process of
    re-replicating that data elsewhere, that can use cluster resources.

    You MIGHT want to avoid that, or maybe you do not care.

    Having 2 disks for the OS is a waist of bays, so we got clever. Take a
    system with 8 drives @ 1TB. Slice off ~30 GB from two of the disks and
    use Linux software RAID-1 MIRROR for the OS+ swap.

    Now you don't need to separate disks for the OS and you don't run the
    risk of losing that one disk that takes down the entire DataNode.
  • Patrick Angeles at Oct 15, 2009 at 4:51 pm
    On Thu, Oct 15, 2009 at 12:32 PM, Edward Capriolo wrote:
    No need for dedicated SATA drives with
    RAID for your OS. Most of that is accessed during boot time so it won't
    contend that much with HDFS.
    You may want to RAID your OS. If you lose a datanode with a large
    volume of data say (8 TB) Hadoop will begin the process of
    re-replicating that data elsewhere, that can use cluster resources.

    You MIGHT want to avoid that, or maybe you do not care.

    Having 2 disks for the OS is a waist of bays, so we got clever. Take a
    system with 8 drives @ 1TB. Slice off ~30 GB from two of the disks and
    use Linux software RAID-1 MIRROR for the OS+ swap.

    Now you don't need to separate disks for the OS and you don't run the
    risk of losing that one disk that takes down the entire DataNode.
    Forgot to mention it, but that is exactly what we do. We considered
    net-booting as an option, but we were time-constrained and so didn't look
    that deeply into it. I'd be interested in hearing others that have used
    networked boot...
  • Edward Capriolo at Oct 15, 2009 at 5:26 pm

    On Thu, Oct 15, 2009 at 12:51 PM, Patrick Angeles wrote:
    On Thu, Oct 15, 2009 at 12:32 PM, Edward Capriolo wrote:

    No need for dedicated SATA drives with
    RAID for your OS. Most of that is accessed during boot time so it won't
    contend that much with HDFS.
    You may want to RAID your OS. If you lose a datanode with a large
    volume of data say (8 TB) Hadoop will begin the process of
    re-replicating that data elsewhere, that can use cluster resources.

    You MIGHT want to avoid that, or maybe you do not care.

    Having 2 disks for the OS is a waist of bays, so we got clever. Take a
    system with 8 drives @ 1TB. Slice off ~30 GB from two of the disks and
    use Linux software RAID-1 MIRROR for the OS+ swap.

    Now you don't need to separate disks for the OS and you don't run the
    risk of losing that one disk that takes down the entire DataNode.
    Forgot to mention it, but that is exactly what we do. We considered
    net-booting as an option, but we were time-constrained and so didn't look
    that deeply into it. I'd be interested in hearing others that have used
    networked boot...
    Patrick,

    I saw someone troubleshooting netboot recently on list. There was some
    minor hangup, I they got it working

    http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200909.mbox/%3C4AC22618.5080701@sci.utah.edu%3E

    Ed

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 15, '09 at 3:48p
activeOct 15, '09 at 7:21p
posts9
users7
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase