FAQ
I am using the following to set my number of reduce tasks, however when I
run my job it's always using just 1 reducer.

conf.setInt("mapred.reduce.tasks", 20);

1 reducer will never finish this job. Please help me to understand why the
setting I choose is not used.

Thanks,
-M@

Search Discussions

  • Matt Tanquary at Oct 21, 2010 at 8:42 pm
    A little more detail:

    I have mapred.tasktracker.reduce.tasks.maximum set to 4 and I have 5 nodes.
    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary wrote:

    I am using the following to set my number of reduce tasks, however when I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand why the
    setting I choose is not used.

    Thanks,
    -M@

    --
    Have you thanked a teacher today? ---> http://www.liftateacher.org
  • Alex Kozlov at Oct 21, 2010 at 8:45 pm
    Hi Matt, it might be that the parameter does not end up in the final
    configuration for a number of reasons. Can you check the job config xml in
    jt:/var/log/hadoop/history or in the JT UI and see what the
    mapred.reduce.tasks setting is? -- Alex K
    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary wrote:

    I am using the following to set my number of reduce tasks, however when I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand why the
    setting I choose is not used.

    Thanks,
    -M@
  • Ed at Oct 21, 2010 at 9:17 pm
    You could also try

    job.setNumReduceTasks(yourNumber);

    ~Ed
    On Thu, Oct 21, 2010 at 4:45 PM, Alex Kozlov wrote:

    Hi Matt, it might be that the parameter does not end up in the final
    configuration for a number of reasons. Can you check the job config xml in
    jt:/var/log/hadoop/history or in the JT UI and see what the
    mapred.reduce.tasks setting is? -- Alex K

    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary <matt.tanquary@gmail.com
    wrote:
    I am using the following to set my number of reduce tasks, however when I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand why the
    setting I choose is not used.

    Thanks,
    -M@
  • Matt Tanquary at Oct 21, 2010 at 9:25 pm
    Hi Alex,

    Yes, I confirmed from those locations that the job is setting the reducers
    to 1.

    Thanks
    On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov wrote:

    Hi Matt, it might be that the parameter does not end up in the final
    configuration for a number of reasons. Can you check the job config xml in
    jt:/var/log/hadoop/history or in the JT UI and see what the
    mapred.reduce.tasks setting is? -- Alex K

    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary <matt.tanquary@gmail.com
    wrote:
    I am using the following to set my number of reduce tasks, however when I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand why the
    setting I choose is not used.

    Thanks,
    -M@


    --
    Have you thanked a teacher today? ---> http://www.liftateacher.org
  • Alex Kozlov at Oct 21, 2010 at 9:33 pm
    It looks like you do not pass the Configuration object correctly. Do you
    use old or new (mapreduce) API? Do you have something like

    Job job = new Job(conf, "My job with " + conf.get("mapred.reduce.tasks") + "
    reducers");

    to create the job? Is it OK to share you job creation code?

    Alex K
    On Thu, Oct 21, 2010 at 2:25 PM, Matt Tanquary wrote:

    Hi Alex,

    Yes, I confirmed from those locations that the job is setting the reducers
    to 1.

    Thanks
    On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov wrote:

    Hi Matt, it might be that the parameter does not end up in the final
    configuration for a number of reasons. Can you check the job config xml in
    jt:/var/log/hadoop/history or in the JT UI and see what the
    mapred.reduce.tasks setting is? -- Alex K

    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary <matt.tanquary@gmail.com
    wrote:
    I am using the following to set my number of reduce tasks, however when
    I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand why the
    setting I choose is not used.

    Thanks,
    -M@


    --
    Have you thanked a teacher today? ---> http://www.liftateacher.org
  • Matt Tanquary at Oct 21, 2010 at 9:39 pm
    I used the new API. I just tried the job.setNumReduceTasks() method and it
    seems to be working.

    Thanks Alex and Ed for helping me!
    On Thu, Oct 21, 2010 at 2:32 PM, Alex Kozlov wrote:

    It looks like you do not pass the Configuration object correctly. Do you
    use old or new (mapreduce) API? Do you have something like

    Job job = new Job(conf, "My job with " + conf.get("mapred.reduce.tasks") +
    "
    reducers");

    to create the job? Is it OK to share you job creation code?

    Alex K

    On Thu, Oct 21, 2010 at 2:25 PM, Matt Tanquary <matt.tanquary@gmail.com
    wrote:
    Hi Alex,

    Yes, I confirmed from those locations that the job is setting the reducers
    to 1.

    Thanks
    On Thu, Oct 21, 2010 at 1:45 PM, Alex Kozlov wrote:

    Hi Matt, it might be that the parameter does not end up in the final
    configuration for a number of reasons. Can you check the job config
    xml
    in
    jt:/var/log/hadoop/history or in the JT UI and see what the
    mapred.reduce.tasks setting is? -- Alex K

    On Thu, Oct 21, 2010 at 1:39 PM, Matt Tanquary <
    matt.tanquary@gmail.com
    wrote:
    I am using the following to set my number of reduce tasks, however
    when
    I
    run my job it's always using just 1 reducer.

    conf.setInt("mapred.reduce.tasks", 20);

    1 reducer will never finish this job. Please help me to understand
    why
    the
    setting I choose is not used.

    Thanks,
    -M@


    --
    Have you thanked a teacher today? ---> http://www.liftateacher.org


    --
    Have you thanked a teacher today? ---> http://www.liftateacher.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 21, '10 at 8:40p
activeOct 21, '10 at 9:39p
posts7
users3
websitehadoop.apache.org...
irc#hadoop

3 users in discussion

Matt Tanquary: 4 posts Alex Kozlov: 2 posts Ed: 1 post

People

Translate

site design / logo © 2022 Grokbase