Hi,
I am trying to run a shell command from within a mapper. The shell command
is of the form:
* hadoop jar somjarfile arg1 arg2 ...*


Can i do this type of operation from within a mapper?

Also, can i copy the jar file in distributed cache and use it?


Thanks,
Souri

Search Discussions

  • Robert Evans at Dec 14, 2011 at 9:09 pm
    Souri,

    Yes and no. Oozie does something like this, but not through the shell, and you can look at how they are doing it. I don't know the details, but you will probably need to get some delegation tokens to make it work properly if you have security enabled. You should be able to copy the jar file over in the distributed cache without any problems. The real question is why are you doing this? Oozie is doing it to avoid security problems with how it runs user code. Is this something to launch more jobs in some sort of a loop? If so you need to be very careful that you don't end up doing a DDOS on the JobTracker by submitting way too many jobs.

    --Bobby Evans

    On 12/14/11 5:24 AM, "souri datta" wrote:

    Hi,
    I am trying to run a shell command from within a mapper. The shell command is of the form:
    hadoop jar somjarfile arg1 arg2 ...


    Can i do this type of operation from within a mapper?

    Also, can i copy the jar file in distributed cache and use it?


    Thanks,
    Souri

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 14, '11 at 11:24a
activeDec 14, '11 at 9:09p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Robert Evans: 1 post Souri datta: 1 post

People

Translate

site design / logo © 2022 Grokbase