FAQ
I want to partition my data by using R reducer tasks to produce R reduce
output files, and each reduce task also writes a binary file for the
corresponding partition on DFS. Is there an easy way to generate
matching file names for the reduce output and the extra file?

For example:

reduce output: part-0 part-1 ... part-<R-1>
extra file: file-0 file-1 ... file-<R-1>

I can't find how to access task id in reduce.

Thanks.

--hao

Search Discussions

  • Owen O'Malley at Aug 8, 2007 at 10:09 pm

    On Aug 5, 2007, at 8:28 PM, Hao Zheng wrote:

    I can't find how to access task id in reduce.
    For now, the best way is to look in the config via conf.get
    ("mapred.task.id").

    It is documented here:
    http://wiki.apache.org/lucene-hadoop/TaskExecutionEnvironment

    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 6, '07 at 3:29a
activeAug 8, '07 at 10:09p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Hao Zheng: 1 post Owen O'Malley: 1 post

People

Translate

site design / logo © 2021 Grokbase