Grokbase Groups Pig user April 2008
FAQ
I tried to figure how Pig set the number of task for Map and Reduce jobs.

The number of Map task is always tied to the number of input file.
Since there is one input file, number of Map tasks is 1, enven I had a
5.4 GB file and more than 1000 blocks.
setting mapred.amp.taks has no effect what so ever.

<property>
<name>mapred.map.tasks</name>
<value>7</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>

The number of Reduce tasks could be set by Hadoop-site.xml
<property>
<name>mapred.reduce.tasks</name>
<value>2</value>
<description>The default number of reduce tasks per job. Typically set
to a prime close to the number of available hosts. Ignored when
mapred.job.tracker is "local".
</description>
</property>

Please advise,

MIckey Hsieh

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 2 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedApr 11, '08 at 10:09p
activeApr 11, '08 at 10:21p
posts2
users2
websitepig.apache.org

2 users in discussion

Mickey hsieh: 1 post Alan Gates: 1 post

People

Translate

site design / logo © 2022 Grokbase