Hello all,
I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.

I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.

Any help would be appreciated.

Thanks
Praveen

Search Discussions

  • Harsh J at Dec 29, 2010 at 3:46 pm
    Add them to the distributed cache, as must be done with any local file
    that needs distribution to Task JVMs.
    DistributedCache.addFileToClassPath(...) should help.

    --
    Harsh J
    www.harshj.com
  • Hiller, Dean (Contractor) at Dec 29, 2010 at 11:03 pm
    Google DistributedCache hadoop...I think there is a method for
    DistributedCache.addResource though not sure if adding just text files
    or jars, but maybe that will help you in looking. Sorry, I am a newb as
    well.

    Dean



    From: praveen.peddi@nokia.com
    Sent: Wednesday, December 29, 2010 8:40 AM
    To: mapreduce-user@hadoop.apache.org
    Subject: External jar dependency of map reduce jobs



    Hello all,

    I have few map reduce jobs that I am invoking from a glasfish container.
    I am not using "hadoop" command line tool but calling the jobs directly
    from glassfish programatically. I have a driver that runs on glassfish
    and calls these jobs sequentially. I was able to run the jobs as long as
    they do not depend on external jar files. But I have one job that
    depends on external jars and that fails with ClassNotFound exception.



    I know that -libjars options can be used when using command line hadoop
    but since I am invoking the jobs programatically from an app server, I
    am wondering how I can add external dependencies in this case.



    Any help would be appreciated.



    Thanks

    Praveen




    This message and any attachments are intended only for the use of the addressee and
    may contain information that is privileged and confidential. If the reader of the
    message is not the intended recipient or an authorized representative of the
    intended recipient, you are hereby notified that any dissemination of this
    communication is strictly prohibited. If you have received this communication in
    error, please notify us immediately by e-mail and delete the message and any
    attachments from your system.
  • Halai, Murtaza at Dec 29, 2010 at 11:25 pm
    Hi,

    In out build.xml we have a target that creates a standalone.jar that contains the class files belonging to the project and also includes the jars for all transitive dependencies. We use this jar to launch our jobs and in the jobConf if you set your mapred.jar to this jar things should work.
    See: https://p4db.amazon.com/source-views/p4db/brazil/fileViewer.cgi?FSPC=//brazil/src/appgroup/duplicateDetection/apps/DarwinInferredVariationsService/mainline/build.xml&REV=10

    Murtaza



    From: Hiller, Dean (Contractor)
    Sent: Wednesday, December 29, 2010 3:03 PM
    To: mapreduce-user@hadoop.apache.org
    Subject: RE: External jar dependency of map reduce jobs

    Google DistributedCache hadoop...I think there is a method for DistributedCache.addResource though not sure if adding just text files or jars, but maybe that will help you in looking. Sorry, I am a newb as well.
    Dean

    From: praveen.peddi@nokia.com
    Sent: Wednesday, December 29, 2010 8:40 AM
    To: mapreduce-user@hadoop.apache.org
    Subject: External jar dependency of map reduce jobs

    Hello all,
    I have few map reduce jobs that I am invoking from a glasfish container. I am not using "hadoop" command line tool but calling the jobs directly from glassfish programatically. I have a driver that runs on glassfish and calls these jobs sequentially. I was able to run the jobs as long as they do not depend on external jar files. But I have one job that depends on external jars and that fails with ClassNotFound exception.

    I know that -libjars options can be used when using command line hadoop but since I am invoking the jobs programatically from an app server, I am wondering how I can add external dependencies in this case.

    Any help would be appreciated.

    Thanks
    Praveen


    This message and any attachments are intended only for the use of the addressee and

    may contain information that is privileged and confidential. If the reader of the

    message is not the intended recipient or an authorized representative of the

    intended recipient, you are hereby notified that any dissemination of this

    communication is strictly prohibited. If you have received this communication in

    error, please notify us immediately by e-mail and delete the message and any

    attachments from your system.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 29, '10 at 3:40p
activeDec 29, '10 at 11:25p
posts4
users4
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase