FAQ
I am writing an application that submits job jar files to the job tracker. The application writes some files to HDFS among other things before triggering the job. I am using the hadoop-core library in the Maven central repository. Unfortunately this library has several dependencies that I don't believe I need for a client application, such as Jasper, Jetty, and such. Is there a list of the jar files needed for developing Hadoop client applications? Or alternatively, a list of jar files only needed when running the Hadoop server processes?

Search Discussions

  • Bryan Keller at Mar 2, 2011 at 6:11 am
    I took a look at the ivy.xml file and it seems for a Hadoop client, you'll need hadoop-core, commons-http-client, commons-codec, and commons-cli. I also found that you will need jackson (used by JobClient), but this is not specified as a client dependency in the ivy.xml.
    On Mar 1, 2011, at 2:33 AM, Bryan Keller wrote:

    I am writing an application that submits job jar files to the job tracker. The application writes some files to HDFS among other things before triggering the job. I am using the hadoop-core library in the Maven central repository. Unfortunately this library has several dependencies that I don't believe I need for a client application, such as Jasper, Jetty, and such. Is there a list of the jar files needed for developing Hadoop client applications? Or alternatively, a list of jar files only needed when running the Hadoop server processes?
  • Owen O'Malley at Mar 2, 2011 at 6:38 am

    On Tue, Mar 1, 2011 at 2:33 AM, Bryan Keller wrote:

    I am writing an application that submits job jar files to the job tracker.
    The application writes some files to HDFS among other things before
    triggering the job. I am using the hadoop-core library in the Maven central
    repository. Unfortunately this library has several dependencies that I don't
    believe I need for a client application, such as Jasper, Jetty, and such. Is
    there a list of the jar files needed for developing Hadoop client
    applications? Or alternatively, a list of jar files only needed when running
    the Hadoop server processes?

    We should filter the dependencies. In theory to get the client jar
    dependencies, you should use:

    % hadoop classpath

    unfortunately, currently that returns both the server and client jars.

    -- Owen

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 1, '11 at 10:34a
activeMar 2, '11 at 6:38a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Bryan Keller: 2 posts Owen O'Malley: 1 post

People

Translate

site design / logo © 2022 Grokbase