FAQ
I have a java app that runs in tomcat and now needs to talk to my
hadoop infrastructure. Typically, all the testing ive done / examples
show starting something that uses hadoop via the 'bin/hadoop -jar'
cmd, but as you can imagine this is no good for a existing tomcat app.
Ive looked thru the .sh files in the bin/ dir, and it would require
extensive work to mod the script to export the env variables so that
tomcat can be restarted without a special init script (and with those
variables in tact). The last thing i want to do is hand crank files
that may or may not change in new hadoop distros. Is there a known way
to use the hadoop infrastructure outside of the bin/hadoop -jar command?

mb

Search Discussions

  • Jason hadoop at Jul 3, 2009 at 2:06 am
    There is a decent section on this in the pro hadoop book, basically part of
    the unit test section.

    You have to define a hadoop.log.dir java property and ensure that the
    hadoop-default.xml file and any local parameters you need are available.
    Then you can run standalone with no particular trouble.
    On Thu, Jul 2, 2009 at 11:33 AM, Michael Basnight wrote:

    I have a java app that runs in tomcat and now needs to talk to my hadoop
    infrastructure. Typically, all the testing ive done / examples show starting
    something that uses hadoop via the 'bin/hadoop -jar' cmd, but as you can
    imagine this is no good for a existing tomcat app. Ive looked thru the .sh
    files in the bin/ dir, and it would require extensive work to mod the script
    to export the env variables so that tomcat can be restarted without a
    special init script (and with those variables in tact). The last thing i
    want to do is hand crank files that may or may not change in new hadoop
    distros. Is there a known way to use the hadoop infrastructure outside of
    the bin/hadoop -jar command?

    mb


    --
    Pro Hadoop, a book to guide you from beginner to hadoop mastery,
    http://www.amazon.com/dp/1430219424?tag=jewlerymall
    www.prohadoopbook.com a community for Hadoop Professionals
  • Steve Loughran at Jul 3, 2009 at 2:35 pm

    Michael Basnight wrote:
    I have a java app that runs in tomcat and now needs to talk to my hadoop
    infrastructure. Typically, all the testing ive done / examples show
    starting something that uses hadoop via the 'bin/hadoop -jar' cmd, but
    as you can imagine this is no good for a existing tomcat app. Ive looked
    thru the .sh files in the bin/ dir, and it would require extensive work
    to mod the script to export the env variables so that tomcat can be
    restarted without a special init script (and with those variables in
    tact). The last thing i want to do is hand crank files that may or may
    not change in new hadoop distros. Is there a known way to use the hadoop
    infrastructure outside of the bin/hadoop -jar command?

    mb

    what are you trying to do? Submit jobs or start hadoop itself?

    Hadoop is tricky to start up in-VM; the hadoop-3628 branch of trunk can
    do this, but your security manager needs to intercept the odd call to
    System.exit(), and there are a lot of singletons for monitoring -better
    to start up hadoop in a new VM.

    1. JobSubmission can be done remotely by way of the JobClient api

    http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/components/submitter/SubmitterImpl.java?view=markup

    2. You can also run any instance of Tool by creating then invoking it
    -with any configuration you choose to run

    http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/components/submitter/ToolRunnerComponentImpl.java?view=markup

    Both suffer from version-sensitivity, everything has to be on exactly
    the same version of hadoop, and the hadoop cluster has to be exposed to
    tomcat, which can be a problem if tomcat is in a DMZ and you are trying
    to secure Hadoop.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 2, '09 at 6:33p
activeJul 3, '09 at 2:35p
posts3
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase