FAQ
Hi,

The following was working fine with Hadoop Writables.
Now, I'm using my custom Writable class called "TermFreqArrayWritable" to produce a Sequence File with key=LongWritable and value=TermFreqArrayWritable.

However, when I try to read the produced Sequence File using its Reader, I get the following:

java.lang.RuntimeException: java.io.IOException: WritableName can't load class: TermFreqArrayWritable
at org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1615)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1417)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFileReader_HadoopJob.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.IOException: WritableName can't load class: TermFreqArrayWritable
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
at org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1613)
... 10 more
Caused by: java.lang.ClassNotFoundException: TermFreqArrayWritable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
... 11 more

By the way, my SequenceFileReader has those lines:

LongWritable key = new LongWritable();
TermFreqArrayWritable value = new TermFreqArrayWritable();
while(reader.next(key,value)){
System.out.println("key: "+ key.toString());
System.out.println("value: "+ value.toString());
}
and TermFreqArrayWritable is inside the same project under a default package.


Has any one tried their custom Writable with SequenceFiles ?

Thank you,
Maha

Search Discussions

  • Simon at Mar 20, 2011 at 2:13 am
    It is hard to judge without the code. But my guess is that your
    TermFreqArrayWritable
    is not properly compiled or imported into your job control file.

    HTH.
    Simon
    On Fri, Mar 18, 2011 at 7:23 PM, maha wrote:

    Hi,

    The following was working fine with Hadoop Writables.
    Now, I'm using my custom Writable class called "TermFreqArrayWritable" to
    produce a Sequence File with key=LongWritable and
    value=TermFreqArrayWritable.

    However, when I try to read the produced Sequence File using its Reader, I
    get the following:

    java.lang.RuntimeException: java.io.IOException: WritableName can't load
    class: TermFreqArrayWritable
    at
    org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1615)
    at
    org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
    at
    SequenceFileReader_HadoopJob.main(SequenceFileReader_HadoopJob.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.io.IOException: WritableName can't load class:
    TermFreqArrayWritable
    at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
    at
    org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1613)
    ... 10 more
    Caused by: java.lang.ClassNotFoundException: TermFreqArrayWritable
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
    org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
    at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
    ... 11 more

    By the way, my SequenceFileReader has those lines:

    LongWritable key = new LongWritable();
    TermFreqArrayWritable value = new TermFreqArrayWritable();
    while(reader.next(key,value)){
    System.out.println("key: "+ key.toString());
    System.out.println("value: "+ value.toString());
    }
    and TermFreqArrayWritable is inside the same project under a default
    package.


    Has any one tried their custom Writable with SequenceFiles ?

    Thank you,
    Maha

    --
    Regards,
    Simon
  • Maha at Mar 20, 2011 at 4:46 am
    That's absolutely correct :) thanks Simon.

    Maha
    On Mar 19, 2011, at 7:13 PM, Simon wrote:

    It is hard to judge without the code. But my guess is that your
    TermFreqArrayWritable
    is not properly compiled or imported into your job control file.

    HTH.
    Simon
    On Fri, Mar 18, 2011 at 7:23 PM, maha wrote:

    Hi,

    The following was working fine with Hadoop Writables.
    Now, I'm using my custom Writable class called "TermFreqArrayWritable" to
    produce a Sequence File with key=LongWritable and
    value=TermFreqArrayWritable.

    However, when I try to read the produced Sequence File using its Reader, I
    get the following:

    java.lang.RuntimeException: java.io.IOException: WritableName can't load
    class: TermFreqArrayWritable
    at
    org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1615)
    at
    org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
    at
    org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
    at
    SequenceFileReader_HadoopJob.main(SequenceFileReader_HadoopJob.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
    Caused by: java.io.IOException: WritableName can't load class:
    TermFreqArrayWritable
    at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
    at
    org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1613)
    ... 10 more
    Caused by: java.lang.ClassNotFoundException: TermFreqArrayWritable
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
    org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
    at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
    ... 11 more

    By the way, my SequenceFileReader has those lines:

    LongWritable key = new LongWritable();
    TermFreqArrayWritable value = new TermFreqArrayWritable();
    while(reader.next(key,value)){
    System.out.println("key: "+ key.toString());
    System.out.println("value: "+ value.toString());
    }
    and TermFreqArrayWritable is inside the same project under a default
    package.


    Has any one tried their custom Writable with SequenceFiles ?

    Thank you,
    Maha

    --
    Regards,
    Simon

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMar 18, '11 at 11:24p
activeMar 20, '11 at 4:46a
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Maha: 2 posts Simon: 1 post

People

Translate

site design / logo © 2022 Grokbase