FAQ
I am trying to create a bundled jar file for running using the hadoop ‘jar’ command. However, when I try to do this it fails to
find the jar files and other resources that I have placed into the jar (pointed at by the Class-Path property in the MANIFEST.MF).

I have unpacked the jar file into a temporary directory and run it manually using the java -jar command (after manually listing
the various hadoop jar files that are required in the classpath) and it works fine in this mode.

I have read on some places that the workaround is to put my jar files and other resources into the hadoop ‘lib’ directory, but
I really don’t want to do this, since it feels horribly kludgy. I don’t mind creating a ‘lib’ directory in my jar file, since this will be
hidden, but I would appreciate more documentation as to why/how I need to do this.

Can anybody advise as to how I can get 'hadoop jar’ to honour the Class-Path entry in the MANIFEST.MF?

Thanks,

Andy Doddington

Search Discussions

  • Steve Loughran at Mar 23, 2011 at 11:44 am

    On 22/03/11 13:34, Andy Doddington wrote:
    I am trying to create a bundled jar file for running using the hadoop ‘jar’ command. However, when I try to do this it fails to
    find the jar files and other resources that I have placed into the jar (pointed at by the Class-Path property in the MANIFEST.MF).

    I have unpacked the jar file into a temporary directory and run it manually using the java -jar command (after manually listing
    the various hadoop jar files that are required in the classpath) and it works fine in this mode.

    I have read on some places that the workaround is to put my jar files and other resources into the hadoop ‘lib’ directory, but
    I really don’t want to do this, since it feels horribly kludgy.
    Putting them in lib/ is bad as you need to restart the cluster to get
    changes out, and you can't have jobs with different versions in them

    use the -libjars option instead
    I don’t mind creating a ‘lib’ directory in my jar file, since this will be
    hidden, but I would appreciate more documentation as to why/how I need to do this.
    because the class-path manifest thing is designed for classloaders in
    the local filesystem where relative paths are used to find JARs and
    everything runs locally, and is only handled by the normal Java main
    entry point and the applet loader.
    Can anybody advise as to how I can get 'hadoop jar’ to honour the Class-Path entry in the MANIFEST.MF?
    You'd probably have to write the code to
    -parse the entry at job submission
    -find the JARs or fail
    -somehow include all these JARs in the list of dependencies for the
    job submission list via a transform to the -libjars command.
    -add the tests for this

    I'm not sure it's worth the effort.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 22, '11 at 1:35p
activeMar 23, '11 at 11:44a
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Steve Loughran: 1 post Andy Doddington: 1 post

People

Translate

site design / logo © 2023 Grokbase