FAQ
Hi,all

I find that when the sub-nodes' hardware configurations are
different, some nodes are strong(more cpus and more memory), others are
weak (leas cpus and less memory), when I run the job, the task are
almost evenly distributed to all the sub-nodes. This makes the weak
nodes pretty slow and a lot of tasks on the weak nodes are killed. This
may lead the whole job processing becoming slow, I am sure, because a
lot of tasks(more than 10 tasks) are processed twice.

Question: How can I configure the hadoop to distribute less
tasks to weak nodes and distribute more tasks to strong nodes?



I configure the strong nodes with

"mapred.tasktracker.map.tasks.maximum=75",

"mapred.map.tasks=60",

"mapred.tasktracker.reduce.tasks.maximum=18",

"mapred.reduce.tasks=15"



and the weak nodes with

"mapred.tasktracker.map.tasks.maximum=60",

"mapred.map.tasks=45",

"mapred.tasktracker.reduce.tasks.maximum=15",

"mapred.reduce.tasks=12"



I have 4 nodes totally. One for name node and job tracker, the others
are for sub-nodes.



Thanks.



Guibin zhang

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 3 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 28, '08 at 3:31a
activeFeb 28, '08 at 3:35p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase