FAQ
Hi there!

We are evaluating a migration from CDH3 to CDH4 + YARN and now we are
locked in an issue with Mapreduce + Hbase.

Our Scheduler currently is the FifoScheduler - right now we don't have a
scenario for multiuser job submission. There are 6 nodes with RS + NN +
Tasks, and we have one regions allocated for each Region Server. I checked
my table files inside datanodes and each node have one copy of each block
of the StoreFile assigned for each Region Server.

When I run a Job with a TableInputMapper/TableInputFormat for this table,
here is what I got when splits are about to be assigned:

2012-07-02 20:03:32,416 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop10.mercador.local to /default-rack

2012-07-02 20:03:32,418 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2012-07-02 20:03:32,418 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop01.mercador.local to /default-rack

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop08.mercador.local to /default-rack

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop09.mercador.local to /default-rack

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2012-07-02 20:03:32,419 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop12.mercador.local to /default-rack

2012-07-02 20:03:32,420 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000004_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2012-07-02 20:03:32,420 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:32,420 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000005_0 TaskAttempt Transitioned from NEW to UNASSIGNED


And that's Ok, because each resolved name corresponds to my Region Server Nodes. And now what is actually driving me crazy:


2012-07-02 20:03:34,415 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 6

2012-07-02 20:03:34,422 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Assign: PendingReduces:0 ScheduledMaps:0 ScheduledReduces:0 AssignedMaps:6 AssignedReduces:0 completedMaps:0 completedReduces:0 containersAllocated:6 containersReleased:0 hostLocalAssigned:2 rackLocalAssigned:4 availableResources(headroom):memory: 52224

2012-07-02 20:03:34,422 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop01.mercador.local to /default-rack

2012-07-02 20:03:34,500 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2012-07-02 20:03:34,500 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:34,501 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000005_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2012-07-02 20:03:34,502 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:34,502 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2012-07-02 20:03:34,503 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:34,503 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2012-07-02 20:03:34,503 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:34,504 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2012-07-02 20:03:34,504 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hadoop11.mercador.local to /default-rack

2012-07-02 20:03:34,505 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1341266378559_0003_m_000004_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED


All my 6 nodes have enough memory and cpu capacity for this job, so I can't understand why only 2 nodes was assigned for this job. Even running 3 concurrent instances of this jobs, I got this same allocation pattern (2 nodes per job).


Any idea of what can be wrong?


Thanks!


Cristofer


P.S.: Sorry for my incomplete previous message!

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedJul 2, '12 at 11:49p
activeJul 2, '12 at 11:49p
posts1
users1
websitecloudera.com
irc#hadoop

1 user in discussion

Cristofer Weber: 1 post

People

Translate

site design / logo © 2022 Grokbase