Grokbase › Groups › Hadoop › common-user › October 2007
FAQ
Hi All,

I am adding two input dir in a job. Both the input dirs have same <Key.class,
Value.class>. Inside the map method i want to know that which pair<key,
value> has come from which input dir. How can i do this ? Any help will be
appreciated..

Regards,
Shaile

Search Discussions

  • Owen O'Malley at Oct 12, 2007 at 3:52 pm ⇧

    On Oct 12, 2007, at 5:51 AM, Shailendra Mudgal wrote:

    I am adding two input dir in a job. Both the input dirs have same
    <Key.class,
    Value.class>. Inside the map method i want to know that which
    pair<key,
    value> has come from which input dir. How can i do this ? Any help
    will be
    appreciated..
    *sigh* We've _almost_ had that feature for a long time now, see
    HADOOP-372.

    The work around is to use the information on:
    http://wiki.apache.org/lucene-hadoop/TaskExecutionEnvironment
    and get the "map.input.file" from the map's JobConf and match against
    the prefix.
  • Benjamin Reed at Oct 12, 2007 at 5:21 pm ⇧
    We do this in Pig by using our own InputSplits.

    ben
    On Friday 12 October 2007, Owen O'Malley wrote:
    On Oct 12, 2007, at 5:51 AM, Shailendra Mudgal wrote:
    I am adding two input dir in a job. Both the input dirs have same
    <Key.class,
    Value.class>. Inside the map method i want to know that which
    pair<key,
    value> has come from which input dir. How can i do this ? Any help
    will be
    appreciated..
    *sigh* We've _almost_ had that feature for a long time now, see
    HADOOP-372.

    The work around is to use the information on:
    http://wiki.apache.org/lucene-hadoop/TaskExecutionEnvironment
    and get the "map.input.file" from the map's JobConf and match against
    the prefix.
  • Ted Dunning at Oct 12, 2007 at 5:56 pm ⇧
    It is also pretty easy to over-ride bits of TextInputFormat to give the file
    as the key instead of the offset.

    On 10/12/07 10:19 AM, "Benjamin Reed" wrote:

    We do this in Pig by using our own InputSplits.

    ben
    On Friday 12 October 2007, Owen O'Malley wrote:
    On Oct 12, 2007, at 5:51 AM, Shailendra Mudgal wrote:
    I am adding two input dir in a job. Both the input dirs have same
    <Key.class,
    Value.class>. Inside the map method i want to know that which
    pair<key,
    value> has come from which input dir. How can i do this ? Any help
    will be
    appreciated..
    *sigh* We've _almost_ had that feature for a long time now, see
    HADOOP-372.

    The work around is to use the information on:
    http://wiki.apache.org/lucene-hadoop/TaskExecutionEnvironment
    and get the "map.input.file" from the map's JobConf and match against
    the prefix.
  • Shailendra Mudgal at Oct 15, 2007 at 4:52 am ⇧
    Hi,

    Thanks for your responses. As all the solutions were good to me, I hv
    decided to use the "map.input.file" from jobconf.

    Thanks & Regards,
    Shaile..
    On 10/12/07, Ted Dunning wrote:



    It is also pretty easy to over-ride bits of TextInputFormat to give the
    file
    as the key instead of the offset.

    On 10/12/07 10:19 AM, "Benjamin Reed" wrote:

    We do this in Pig by using our own InputSplits.

    ben
    On Friday 12 October 2007, Owen O'Malley wrote:
    On Oct 12, 2007, at 5:51 AM, Shailendra Mudgal wrote:
    I am adding two input dir in a job. Both the input dirs have same
    <Key.class,
    Value.class>. Inside the map method i want to know that which
    pair<key,
    value> has come from which input dir. How can i do this ? Any help
    will be
    appreciated..
    *sigh* We've _almost_ had that feature for a long time now, see
    HADOOP-372.

    The work around is to use the information on:
    http://wiki.apache.org/lucene-hadoop/TaskExecutionEnvironment
    and get the "map.input.file" from the map's JobConf and match against
    the prefix.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedOct 12, '07 at 12:52p
activeOct 15, '07 at 4:52a
posts5
users4
websitehadoop.apache.org...
irc#hadoop

4 users in discussion

Shailendra Mudgal: 2 posts Benjamin Reed: 1 post Ted Dunning: 1 post Owen O'Malley: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase