FAQ
Im using a custom Writable in a proxy server feeding records to
hadoop.mapred.RecordReaders. The proxy talks hadoop rpc with the record
reader.

This works fine when I run with mapred.job.tracker="local", but I get a
ClassNotFoundException when i run distributed:

282106-2006-08-17 15:02:34,609 INFO org.apache.hadoop.ipc.Client: Client
connection to 127.0.0.1:19093 caught: java.lang.RuntimeException:
readObject can't find class
282266-java.lang.RuntimeException: readObject can't find class
282322- at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:178)
282398- at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
282473- at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)
282538:Caused by: java.lang.ClassNotFoundException:
dk.statsbiblioteket.sbandex.mapred.WritableRecord
282633- at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
282692- at java.security.AccessController.doPrivileged(Native Method)
282755- at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
282818- at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
282876- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
282942- at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
283000- at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
283066- at java.lang.Class.forName0(Native Method)
283110- at java.lang.Class.forName(Class.java:242)
283154- at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:174)
283230- ... 2 more


Do I have to add the module containing
dk.statsbiblioteket.sbandex.mapred.WritableRecord to Hadoops classpath?
I would assume that "hadoop jar myPackage.jar" would import everything
from myPackage.jar...

Cheers,
Mikkel

Search Discussions

  • Benjamin Reed at Aug 18, 2006 at 10:04 pm
    Are you constructing the JobConf with a Class? And is your class file in
    a directory on the classpath (rather than the Jar file)? If it is, your
    jar file will NOT be submitted with the job. (I opened a bug on this a
    while ago.)

    The easiest way to make sure your jar gets sent is to use the setJar()
    method on JobConf.

    ben

    Mikkel Kamstrup Erlandsen wrote:
    Im using a custom Writable in a proxy server feeding records to
    hadoop.mapred.RecordReaders. The proxy talks hadoop rpc with the record
    reader.

    This works fine when I run with mapred.job.tracker="local", but I get a
    ClassNotFoundException when i run distributed:

    282106-2006-08-17 15:02:34,609 INFO org.apache.hadoop.ipc.Client: Client
    connection to 127.0.0.1:19093 caught: java.lang.RuntimeException:
    readObject can't find class
    282266-java.lang.RuntimeException: readObject can't find class
    282322- at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:178)
    282398- at
    org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
    282473- at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)
    282538:Caused by: java.lang.ClassNotFoundException:
    dk.statsbiblioteket.sbandex.mapred.WritableRecord
    282633- at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    282692- at java.security.AccessController.doPrivileged(Native Method)
    282755- at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    282818- at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    282876- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
    282942- at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
    283000- at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
    283066- at java.lang.Class.forName0(Native Method)
    283110- at java.lang.Class.forName(Class.java:242)
    283154- at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:174)
    283230- ... 2 more


    Do I have to add the module containing
    dk.statsbiblioteket.sbandex.mapred.WritableRecord to Hadoops classpath?
    I would assume that "hadoop jar myPackage.jar" would import everything
    from myPackage.jar...

    Cheers,
    Mikkel
  • Mikkel Kamstrup Erlandsen at Aug 24, 2006 at 1:49 pm

    On Fri, 2006-08-18 at 15:03 -0700, Benjamin Reed wrote:
    Are you constructing the JobConf with a Class?
    Yeah, I subclass JobConf, and instantiate that subclass in my jars main.
    And is your class file in
    a directory on the classpath (rather than the Jar file)? If it is, your
    jar file will NOT be submitted with the job. (I opened a bug on this a
    while ago.)
    The jar file which also contains my JobConf subclass was not in the
    class path. However I symlinked it in hadoop/lib/ and I was able to get
    a little further.
    The easiest way to make sure your jar gets sent is to use the setJar()
    method on JobConf.
    I use setJar to point the job at the absolute path of the jar.

    More or less I have this working now. My problems are in another area
    now :-)



    Mikkel Kamstrup Erlandsen wrote:
    Im using a custom Writable in a proxy server feeding records to
    hadoop.mapred.RecordReaders. The proxy talks hadoop rpc with the record
    reader.

    This works fine when I run with mapred.job.tracker="local", but I get a
    ClassNotFoundException when i run distributed:

    282106-2006-08-17 15:02:34,609 INFO org.apache.hadoop.ipc.Client: Client
    connection to 127.0.0.1:19093 caught: java.lang.RuntimeException:
    readObject can't find class
    282266-java.lang.RuntimeException: readObject can't find class
    282322- at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:178)
    282398- at
    org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:60)
    282473- at org.apache.hadoop.ipc.Client$Connection.run(Client.java:170)
    282538:Caused by: java.lang.ClassNotFoundException:
    dk.statsbiblioteket.sbandex.mapred.WritableRecord
    282633- at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    282692- at java.security.AccessController.doPrivileged(Native Method)
    282755- at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    282818- at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    282876- at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
    282942- at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
    283000- at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
    283066- at java.lang.Class.forName0(Native Method)
    283110- at java.lang.Class.forName(Class.java:242)
    283154- at
    org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:174)
    283230- ... 2 more


    Do I have to add the module containing
    dk.statsbiblioteket.sbandex.mapred.WritableRecord to Hadoops classpath?
    I would assume that "hadoop jar myPackage.jar" would import everything
    from myPackage.jar...

    Cheers,
    Mikkel

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedAug 18, '06 at 8:47a
activeAug 24, '06 at 1:49p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase