2010/9/6 褚 鵬兵 <firstname.lastname@example.org>:
hi ,my hadoop friends:i have the 3 questions about hadoop.there are ....
1 the speed between the datanodes. Tera data in one datanodes , the data transfers from one datanode to the another datanode. if the speed is bad, Hadoop will be slow, i think. i heard the gNet architecture in Greenplum , then hadoop ? SAS storage + G-Ethernet is best answer, isn't it?
2 the GUI tool there is a hive web tool in hadoop. but it is not enough to use it for our business work. it is too simple to use it.
if hadoop+hive is designed into DWH. then how to use it for users. by CGI Tool(Command),? by New Developed webGUITOOL.?
3 5 computers Hadoop cluster and 1 computer SQLSERVER2000 5 computers Hadoop celeron 2.66G 1G memory Ethernet namenode + secondarynamenode + 3 datanode 1 computer SQLSERVER2000 celeron 2.66G 1G memory then i did select operation at the same data 100M . 5 computers Hadoop is 2mins 30secs 1 computer SQLSERVER2000 is 2mins 25secs
the result is that 5 computers Hadoop is not good .why .can anyone give me some advises.
thanks in adverse.
Why use Hadoop in preference to a database?
At the recent Hadoop User Group (UK) meeting, Andy Kemp fromhttp://www.forward.co.uk/
presented their experience in moving from a
MySQL database approach to Hadoop.
From my notes of his talk their system manages 120 million keywords
and is updated at a rate of 20GB/day.
They originally used a sharded MySQL database but found it couldn't
scale to handle the types of queries their users required, e.g. "Can
you cluster 17(?) million keyword phrases into thematic groups?".
Their calculations indicated that the database approach would take
more than a year to handle such a query.
Moving to a cluster of 100 Hadoop nodes on Amazon EC2 reduced this
time down to 7 hours. The issues then became one of the costs of
storage and moving the data to and from the cluster.
They then moved to a private VM system with about 30 VMs - I assume
the processing took the same time as I didn't note this down.
From there they then moved to dedicated hardware, 5 dedicated Hadoop
nodes, and achieved better performance than the 30 VMs.
Andy's talk, "Hadoop in Context" should available as a podcast herehttp://skillsmatter.com/podcast/cloud-grid/hadoop-in-context
be well worth watching but when I lasted looked it hadn't been
At the same event, Ian Broadhead, from http://www.playfish.com/
talk on managing the activity of over 1 million active Internet gamers
producing over 50GB of data a day. Their original MySQL system took
up to 50 times longer to process their data load than an EC2 cluster
of Hadoop nodes. He talked about a typical workload being reduced
from 2-3 days (using MySQL) down to 6 hours (using Hadoop).
Unfortunately I don't think Ian's talk will appear as a podcast.
However, most presentations during the evening made a point that
Hadoop didn't completely replace their databases, just provided a
convenient way to rapidly process large volumes of data, the output
from Hadoop processing typically being stored in databases to satisfy
general everyday business queries.
I think the common theme here was that all of these users had large
datasets of the order of 100's of GBs with multiple views of that data
that handled in the order of 10's of millions of updates a day.
I hope that helps.