FAQ
Does Hadoop run multiple threads on a single slave? In my current
experiment, I'm running with three slaves, each of which has four
processors. Hadoop only appears to be using one processor on each
slave. It's very possible that I've made a configuration mistake.

I have mapred.map.tasks set to 7 (this should be enough to see two jobs
on each client, though I plan on eventually setting it higher).
mapred.tasktracker.tasks.maximum is 3 (eventually this will be higher,
too).

Thanks.

--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

Search Discussions

  • Doug Cutting at Mar 22, 2006 at 11:14 pm

    Andrew McNabb wrote:
    Does Hadoop run multiple threads on a single slave?
    Yes, although each task runs in a separate JVM, not a thread. A slave
    will run up to mapred.tasktracker.tasks.maximum map and/or reduce tasks
    at a time.
    I have mapred.map.tasks set to 7 (this should be enough to see two jobs
    on each client, though I plan on eventually setting it higher).
    mapred.tasktracker.tasks.maximum is 3 (eventually this will be higher,
    too).
    The actual number of map tasks is determined by the number of input
    splits. Perhaps your input data is not big enough to result in more
    than a few input splits? A SequenceFile-format input cannot be split
    into chunks smaller than 2k bytes.

    Doug
  • Andrew McNabb at Mar 23, 2006 at 12:06 am

    On Wed, Mar 22, 2006 at 03:13:40PM -0800, Doug Cutting wrote:
    Yes, although each task runs in a separate JVM, not a thread. A slave
    will run up to mapred.tasktracker.tasks.maximum map and/or reduce
    tasks at a time.
    That makes sense.
    The actual number of map tasks is determined by the number of input
    splits. Perhaps your input data is not big enough to result in more
    than a few input splits? A SequenceFile-format input cannot be split
    into chunks smaller than 2k bytes.
    It turns out that the expected number of jobs did run on each
    machine--they just finished quickly enough that I didn't see them in
    top.

    Thanks.

    --
    Andrew McNabb
    http://www.mcnabbs.org/andrew/
    PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 22, '06 at 10:53p
activeMar 23, '06 at 12:06a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Andrew McNabb: 2 posts Doug Cutting: 1 post

People

Translate

site design / logo © 2022 Grokbase