Hello,
Another newbie question.
Suppose I want to use an external library (jar) in the mapper / reducer
classes.
(commons-lang, google's guava, etc.)
In our environment, I added the jars into a specific folder and added them
to HADOOP-CLASSPATH.
However, when running mapper that uses one of the jars, it could not find
the classes in that jar.

I thought that it might be in our environment (I am not managing our
cluster).

Then I read about DistributedCache.
Should I use it with methods such as addArchiveToClassPath,
addFileToClassPath, addCachArchive to use jar libraries?
If so, which method is more appropriate ?

If not, how do we load jar libraries to each VM?

Thanks,

Eyal Golan
egolan74@gmail.com

Visit: http://jvdrums.sourceforge.net/
LinkedIn: http://www.linkedin.com/in/egolan74
Skype: egolan74

P Save a tree. Please don't print this e-mail unless it's really necessary

Search Discussions

  • Praveen Sripati at Dec 28, 2011 at 11:36 am
    Check this article from Cloudera on different ways of distributing a jar
    file to the job.

    http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

    Praveen
    On Wed, Dec 28, 2011 at 5:40 AM, Eyal Golan wrote:

    Hello,
    Another newbie question.
    Suppose I want to use an external library (jar) in the mapper / reducer
    classes.
    (commons-lang, google's guava, etc.)
    In our environment, I added the jars into a specific folder and added them
    to HADOOP-CLASSPATH.
    However, when running mapper that uses one of the jars, it could not find
    the classes in that jar.

    I thought that it might be in our environment (I am not managing our
    cluster).

    Then I read about DistributedCache.
    Should I use it with methods such as addArchiveToClassPath,
    addFileToClassPath, addCachArchive to use jar libraries?
    If so, which method is more appropriate ?

    If not, how do we load jar libraries to each VM?

    Thanks,

    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary
  • Eyal Golan at Dec 28, 2011 at 11:43 am
    great !!

    this is exactly what I was looking for.


    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really necessary



    On Wed, Dec 28, 2011 at 1:35 PM, Praveen Sripati
    wrote:
    Check this article from Cloudera on different ways of distributing a jar
    file to the job.


    http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

    Praveen
    On Wed, Dec 28, 2011 at 5:40 AM, Eyal Golan wrote:

    Hello,
    Another newbie question.
    Suppose I want to use an external library (jar) in the mapper / reducer
    classes.
    (commons-lang, google's guava, etc.)
    In our environment, I added the jars into a specific folder and added
    them to HADOOP-CLASSPATH.
    However, when running mapper that uses one of the jars, it could not find
    the classes in that jar.

    I thought that it might be in our environment (I am not managing our
    cluster).

    Then I read about DistributedCache.
    Should I use it with methods such as addArchiveToClassPath,
    addFileToClassPath, addCachArchive to use jar libraries?
    If so, which method is more appropriate ?

    If not, how do we load jar libraries to each VM?

    Thanks,

    Eyal Golan
    egolan74@gmail.com

    Visit: http://jvdrums.sourceforge.net/
    LinkedIn: http://www.linkedin.com/in/egolan74
    Skype: egolan74

    P Save a tree. Please don't print this e-mail unless it's really
    necessary

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 28, '11 at 12:10a
activeDec 28, '11 at 11:43a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Eyal Golan: 2 posts Praveen Sripati: 1 post

People

Translate

site design / logo © 2022 Grokbase