I was referring to multiple VM's on a single machine (that you have in house) for my previous comment and not EC2. FWIW, I would rather see a single heavy data node than to partition off a single box into multiple machines unless you are trying to do more on that server than just Hadoop. Obviously every person / company has their own constraints but if this box is solely for Hadoop then don't partition it otherwise you will incur a decent loss in possible map/reduce slots.
From: Liam Friel
Sent: Monday, August 15, 2011 3:04 PM
Subject: Re: hadoop cluster on VM's
On Mon, Aug 15, 2011 at 7:31 PM, GOEKE, MATTHEW (AG/1000) wrote:
Is this just for testing purposes or are you planning on going into
production with this? If it is the latter than I would STRONGLY advise to
not give that a second thought due to how the framework handles I/O. However
if you are just trying to test out distributed daemon setup and get some ops
documentation then have at it :)
From: Travis Camechis
Sent: Monday, August 15, 2011 12:45 PM
Subject: hadoop cluster on VM's
Is it recommended to install a hadoop cluster on a set of VM's that are all
connected to a SAN?
Could you expand on that? Do you mean multiple VMs on a single server are a
Or do you mean running Hadoop on something like Amazon EC2 for production is
also a no-no?
With some pointers to background if the latter please ...
Just for my education. I have run some (test I guess you could call them)
Hadoop clusters on EC2 and it was working OK.
However I didn't have the equivalent pile of physical hardware lying around
to do a comparison ... which I guess is why EC2 is so attractive.
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.