FAQ
Hi,



I've built a Hadoop cluster from two computers( master and slave), using
Hadoop 0.18.2/HBase 0.18.1, if I use big files as input( 5-10 GB)
reduce-copy tasks from master to slave is taking too mush time( ~30 minutes)
with speed about 0.10 MB/s, despite the fact that master is connected to
slave via 1GB switch, and I did /etc/hosts mapping using LAN
addresses(10.x.x.x), is there is a way to force hadoop to use ftp for
example for copy of files? Or there is some configuration to improve copy
files performance?



I've tried to copy files with ftp and it works with average speed 50Mb/s.


From reduce task lists:
reduce > copy (67 of 69 at 0.89 MB/s) > : task on master

reduce > copy (29 of 69 at 0.10 MB/s) > : task on slave





Genady

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedDec 26, '08 at 9:04p
activeDec 26, '08 at 9:04p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Genady: 1 post

People

Translate

site design / logo © 2022 Grokbase