[Hadoop-common-user] WritableName can't load class ... for custom WritableClasses

On Fri, Mar 18, 2011 at 7:23 PM, maha wrote:

Hi,

The following was working fine with Hadoop Writables.
Now, I'm using my custom Writable class called "TermFreqArrayWritable" to
produce a Sequence File with key=LongWritable and
value=TermFreqArrayWritable.

However, when I try to read the produced Sequence File using its Reader, I
get the following:

java.lang.RuntimeException: java.io.IOException: WritableName can't load
class: TermFreqArrayWritable
at
org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1615)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
at
SequenceFileReader_HadoopJob.main(SequenceFileReader_HadoopJob.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.IOException: WritableName can't load class:
TermFreqArrayWritable
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
at
org.apache.hadoop.io.SequenceFile$Reader.getValueClass(SequenceFile.java:1613)
... 10 more
Caused by: java.lang.ClassNotFoundException: TermFreqArrayWritable
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
... 11 more

By the way, my SequenceFileReader has those lines:

LongWritable key = new LongWritable();
TermFreqArrayWritable value = new TermFreqArrayWritable();
while(reader.next(key,value)){
System.out.println("key: "+ key.toString());
System.out.println("value: "+ value.toString());
}
and TermFreqArrayWritable is inside the same project under a default
package.


Has any one tried their custom Writable with SequenceFiles ?

Thank you,
Maha

On Mar 19, 2011, at 7:13 PM, Simon wrote:

It is hard to judge without the code. But my guess is that your
TermFreqArrayWritable
is not properly compiled or imported into your job control file.

HTH.
Simon

--
Regards,
Simon