FAQ
hi,
I have 8 machines for the hadoop cluster, 1 namenode and 7 data node.
I want the production jobs to have more priority than the user-defined jobs,
so I use the Fair scheduler.

Why sometimes my job scheduled by user: hadoop just start 7 map tasks, while
in the cluster there's no other job running or waiting to be run?

Here's my configuration section in mapred-site.xml:
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.FairScheduler</value>
</property>
<property>
<name>mapred.fairscheduler.allocation.file</name>
<value>conf/pools.xml</value>
</property>

and here's the pool.xml configuration:
<?xml version="1.0"?>
<allocations>
<pool name="hadoop">
<minMaps>30</minMaps>
<minReduces>30</minReduces>
<weight>4.0</weight>
</pool>
<user name="hive">
<maxRunningJobs>20</maxRunningJobs>
</user>
<userMaxJobsDefault>10</userMaxJobsDefault>
</allocations>

thanks!



--
haitao.yao@Beijing

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedApr 26, '11 at 4:21p
activeApr 26, '11 at 4:21p
posts1
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Erix Yao: 1 post

People

Translate

site design / logo © 2021 Grokbase