FAQ
Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
-------------------------------------------------------------------

Key: HADOOP-189
URL: http://issues.apache.org/jira/browse/HADOOP-189
Project: Hadoop
Type: New Feature

Components: mapred
Reporter: stack@archive.org


Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira

Search Discussions

  • Doug Cutting (JIRA) at May 2, 2006 at 9:54 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting updated HADOOP-189:
    --------------------------------

    Attachment: local-jar.patch

    Tell me if this fixes things for you. Thanks!
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • stack@archive.org (JIRA) at May 3, 2006 at 2:14 am
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=comments#action_12377506 ]

    stack@archive.org commented on HADOOP-189:
    ------------------------------------------

    The patch seems to have no effect. When my map goes to run, unless the classes I want to instantiate are in the root of the job jar, they are not found (ClassNotFoundException). Nothing under the job jar /lib nor /classes directory is ever found.

    Tracing, I see that any new class will have null for ClassLoader -- i.e. the "bootstrap class loader" -- never the URLClassLoader created by this patch.

    Seems like this patch runs too late in the game to be of use? In order to create a job, I need to be able to refer to the implementing class (and possibly configurations) BEFORE LocalJobRunner has had a chance to do its job jar unrolling machinations. To be able to refer to implementing classes, they need to be on the immediate CLASSPATH.

    Seems like any messings w/ classloading populating the CLASSPATH needs to happen before we run Main-Class. This would be most useful I'd imagine. Would a java program that did nothing but read hadoop config. for hadoop local and system dir and then unrolled the job jar finishing by emitting Main-Class and CLASSPATH additions for the hadoop script to exec online #156 be too ugly? It would have the advantage of resemblng how job jars are unrolled out on slaves.

    I can work up such a patch if amenable.


    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 3, 2006 at 3:06 am
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=comments#action_12377510 ]

    Doug Cutting commented on HADOOP-189:
    -------------------------------------
    Tracing, I see that any new class will have null for ClassLoader
    For this patch to work your, e.g., Mapper class needs to only be in the job jar, not also on the classpath built by bin/hadoop. If you are able to instantiate classes in the job jar, and they show up with the bootstrap class loader, then that means the jar file is somehow on the bootstrap classpath, which it should not be after the changes to 'bin/hadoop jar' made in the patch. Is that how you are starting things?

    Another potential problem could be that the URLClassloader is not correctly constructed. The urls added must be either a jar file or end in slashes. It might be useful to log the urls added to the path in LocalJobRunner to check this.
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 3, 2006 at 8:12 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting updated HADOOP-189:
    --------------------------------

    Attachment: local-jar2.patch

    Here's a patch I've actually tested!
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar.patch, local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Michael Stack at May 4, 2006 at 1:53 am
    FYI, the final commit on HADOOP-189 was failing for me in really weird
    ways. Compiling hadoop with 1.6.0beta2 looks to be the culprit.

    Just starting up daemons in pseudo-distributed mode, usually NameNode
    and TaskTracker would fail with exceptions like the following:

    060503 180738 12 Client connection to 127.0.0.1:9000 caught:
    java.lang.RuntimeException: java.lang.ClassNotFoundException:
    [Lorg.apache.hadoop.dfs.Block;
    java.lang.RuntimeException: java.lang.ClassNotFoundException:
    [Lorg.apache.hadoop.dfs.Block;
    at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:177)
    at
    org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)


    Or...

    060503 175932 Client connection to 127.0.0.1:9001 caught:
    java.lang.RuntimeException: java.lang.ClassNotFoundException:
    [Ljava.lang.String;
    java.lang.RuntimeException: java.lang.ClassNotFoundException:
    [Ljava.lang.String;
    at
    org.apache.hadoop.io.ObjectWritable$NullInstance.readFields(ObjectWritable.java:94)
    at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:230)
    at
    org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)

    I'd compiled hadoop with beta2 1.6.0:

    [bregeon 1506] hadoop > /usr/local/jdk1.6.0/bin/java -version
    java version "1.6.0-beta2"
    Java(TM) SE Runtime Environment (build 1.6.0-beta2-b76)
    Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-beta2-b76, mixed mode)


    After recompiling with 1.5.0_06, the above issues went away.

    St.Ack





    Doug Cutting (JIRA) wrote:
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting updated HADOOP-189:
    --------------------------------

    Attachment: local-jar2.patch

    Here's a patch I've actually tested!

    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar.patch, local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
  • Doug Cutting (JIRA) at May 3, 2006 at 8:21 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting updated HADOOP-189:
    --------------------------------

    Attachment: (was: local-jar.patch)
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • stack@archive.org (JIRA) at May 3, 2006 at 8:38 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=comments#action_12377624 ]

    stack@archive.org commented on HADOOP-189:
    ------------------------------------------

    Ok. This patch works really well (Looks to be spin on unbundling before invocation of Main-Class). Thanks for all your work on this.
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Attachments: local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 3, 2006 at 8:50 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting resolved HADOOP-189:
    ---------------------------------

    Fix Version: 0.2
    Resolution: Fixed

    I just committed this.
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Fix For: 0.2
    Attachments: local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 3, 2006 at 10:27 pm
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting reopened HADOOP-189:
    ---------------------------------

    Assign To: Doug Cutting

    I just reverted this, since it broke distributed operation.
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Assignee: Doug Cutting
    Fix For: 0.2
    Attachments: local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira
  • Doug Cutting (JIRA) at May 4, 2006 at 12:14 am
    [ http://issues.apache.org/jira/browse/HADOOP-189?page=all ]

    Doug Cutting resolved HADOOP-189:
    ---------------------------------

    Resolution: Fixed

    Okay, this time it works in both standalone and distributed modes.
    Add job jar lib, classes, etc. to CLASSPATH when in standalone mode
    -------------------------------------------------------------------

    Key: HADOOP-189
    URL: http://issues.apache.org/jira/browse/HADOOP-189
    Project: Hadoop
    Type: New Feature
    Components: mapred
    Reporter: stack@archive.org
    Assignee: Doug Cutting
    Fix For: 0.2
    Attachments: local-jar2.patch

    Currently, in standalone mode, hadoop is unable to launch other than the most basic of job jars where 'basic' is a job jar with nought but class files at top level of the jar with Main-Class pointing at entry point. If the job jar has dependencies on jars under the job jar lib or there are job jar plugins in the classes dir, etc., these dependencies are not loaded and the job fails launch.
    --
    This message is automatically generated by JIRA.
    -
    If you think it was sent incorrectly contact one of the administrators:
    http://issues.apache.org/jira/secure/Administrators.jspa
    -
    For more information on JIRA, see:
    http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMay 2, '06 at 5:45p
activeMay 4, '06 at 1:53a
posts11
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase