FAQ
Would anybody help me?
Can I use
-jobconf mapred.map.task=50 in streaming command to change the job's number of mappers?

I don't have a hadoop at hand and can not verify it. Thanks for your help.

--- On Wed, 10/15/08, Steve Gao wrote:
From: Steve Gao <steve.gao@yahoo.com>
Subject: How to change number of mappers in Hadoop streaming?
To: core-user@hadoop.apache.org
Cc: core-dev@hadoop.apache.org
Date: Wednesday, October 15, 2008, 7:25 PM

Is there a way to change number of mappers in Hadoop streaming command line?
I know I can change hadoop-default.xml:

<property>
<name>mapred.map.tasks</name>
<value>10</value>
<description>The default number of map tasks per job.  Typically set
to a prime several times greater than number of available hosts.
Ignored when mapred.job.tracker is "local".
</description>
</property>

But that's for all jobs. What if I just want each job has different
NUM_OF_Mappers themselves? Thanks

Search Discussions

  • Chaitanya krishna at Oct 26, 2008 at 11:17 am
    Hi,

    In order to have different number of map tasks for each of the jobs, in the
    run method of the code , I had the following syntax:

    conf.setNumMapTasks(num); // for number of map tasks

    conf.setNumReduceTasks(num); // for number of reduce tasks

    conf is the JobConf object and num is the number of map tasks/reduce tasks
    that are to be had for the job. It worked for me, hope it works for you. :)

    btw, i'm using hadoop-0.17.0. I'm not sure if there is any deprecation of
    these methods in later versions.


    V.V.Chaitanya Krishna
    IIIT,Hyderabad
    India
    On Fri, Oct 17, 2008 at 1:25 AM, Steve Gao wrote:


    Would anybody help me?
    Can I use
    -jobconf mapred.map.task=50 in streaming command to change the job's number
    of mappers?

    I don't have a hadoop at hand and can not verify it. Thanks for your help.

    --- On Wed, 10/15/08, Steve Gao wrote:
    From: Steve Gao <steve.gao@yahoo.com>
    Subject: How to change number of mappers in Hadoop streaming?
    To: core-user@hadoop.apache.org
    Cc: core-dev@hadoop.apache.org
    Date: Wednesday, October 15, 2008, 7:25 PM

    Is there a way to change number of mappers in Hadoop streaming command
    line?
    I know I can change hadoop-default.xml:

    <property>
    <name>mapred.map.tasks</name>
    <value>10</value>
    <description>The default number of map tasks per job. Typically set
    to a prime several times greater than number of available hosts.
    Ignored when mapred.job.tracker is "local".
    </description>
    </property>

    But that's for all jobs. What if I just want each job has different
    NUM_OF_Mappers themselves? Thanks






Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedOct 16, '08 at 7:55p
activeOct 26, '08 at 11:17a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Chaitanya krishna: 1 post Steve Gao: 1 post

People

Translate

site design / logo © 2022 Grokbase