[Hadoop-common-user] hadoop index org/apache/lucene/store/Directory NoClassDefFoundError

[Hadoop-common-user] hadoop index org/apache/lucene/store/Directory NoClassDefFoundError

	Chenlbspace
	Oct 10, 2008 at 8:36 am

hi,

I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.

How solve? Thinks.

Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"

---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more

-----------------------------------------------------------------------------------------------------------------

2008-10-10

chenlbspace

Search Discussions

6 responses
Oldest
Nested

Imcaptor U must download the lucene binary jar file,and set the jar file in the classpath. chenlbspace 写道:

Imcaptor at Oct 10, 2008 at 8:39 am ⇧

U must download the lucene binary jar file,and set the jar file in the
classpath.

chenlbspace 写道:
hi,

I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.

How solve? Thinks.

Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"

---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more

-----------------------------------------------------------------------------------------------------------------

2008-10-10

chenlbspace

reply | permalink
Imcaptor The jar file is in the src/contrib/index/lib/lucene-core-2.3.1.jar, u must set the file in classpath. imcaptor 写道:

Imcaptor at Oct 10, 2008 at 8:55 am ⇧

The jar file is in the src/contrib/index/lib/lucene-core-2.3.1.jar, u
must set the file in classpath.

imcaptor 写道:
U must download the lucene binary jar file,and set the jar file in the
classpath.

chenlbspace 写道:
hi,

I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.

How solve? Thinks.

Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"

---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more

-----------------------------------------------------------------------------------------------------------------

2008-10-10

chenlbspace

reply | permalink
Imcaptor Try do this. In the hadoop path, -bash-3.00$ pwd /data/hadoop/hadoop-0.18.1 cp src/contrib/index/lib/lucene-core-2.3.1.jar lib Then you can run the task. chenlbspace 写道:

Imcaptor at Oct 10, 2008 at 9:08 am ⇧

Try do this.

In the hadoop path,
-bash-3.00$ pwd
/data/hadoop/hadoop-0.18.1
cp src/contrib/index/lib/lucene-core-2.3.1.jar lib

Then you can run the task.

chenlbspace 写道:
hi,

I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.

How solve? Thinks.

Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"

---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more

-----------------------------------------------------------------------------------------------------------------

2008-10-10

chenlbspace

reply | permalink

Chenlbspace

at Oct 10, 2008 at 9:50 am

⇧

hi, imcaptor

I try to cp lucene-core-2.3.1.jar to lib, but error also at map task.

-------------------lib-----------------
[[email protected] hadoop]$ pwd
/home/chenlb/hadoop
[[email protected] hadoop]$ ls lib
commons-cli-2.0-SNAPSHOT.jar commons-logging-1.0.4.jar jetty-5.1.4.jar junit-3.8.1.jar kfs-0.1.LICENSE.txt native
commons-codec-1.3.jar commons-logging-api-1.0.4.jar jetty-5.1.4.LICENSE.txt junit-3.8.1.LICENSE.txt log4j-1.2.13.jar servlet-api.jar
commons-httpclient-3.0.1.jar jets3t-0.5.0.jar jetty-ext kfs-0.1.jar lucene-core-2.3.1.jar xmlenc-0.52.jar

-------------------hadoop-site.xml---------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000/</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://master:9001/</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/chenlb/hadoop/tmp/</value>
</property>
</configuration>
----------------------------

Thinks!

=====================error=======================
[[email protected] hadoop]$ bin/hadoop jar hadoop-0.17.1-index.jar -inputPaths data-in -outputPath index-msg-out -indexPath index -conf index-config.xml
08/10/09 22:33:36 INFO main.UpdateIndex: inputPaths = data-in
08/10/09 22:33:36 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 22:33:36 INFO main.UpdateIndex: shards = null
08/10/09 22:33:36 INFO main.UpdateIndex: indexPath = index
08/10/09 22:33:36 INFO main.UpdateIndex: numShards = -1
08/10/09 22:33:36 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 22:33:36 INFO main.UpdateIndex: confPath = index-config.xml
08/10/09 22:33:37 INFO main.UpdateIndex: sea.index.updater = org.apache.hadoop.contrib.index.mapred.IndexUpdater
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.dir = hdfs://master:9000/user/chenlb/data-in
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.output.dir = hdfs://master:9000/user/chenlb/index-msg-out
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.map.tasks = 2
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.reduce.tasks = 1
08/10/09 22:33:37 INFO mapred.IndexUpdater: 1 shards = [email protected]/[email protected]
08/10/09 22:33:37 INFO mapred.IndexUpdater: mapred.input.format.class = org.apache.hadoop.contrib.index.example.LineDocInputFormat
08/10/09 22:33:38 INFO mapred.FileInputFormat: Total input paths to process : 2
08/10/09 22:33:39 INFO mapred.JobClient: Running job: job_200810092201_0010
08/10/09 22:33:40 INFO mapred.JobClient: map 0% reduce 0%
08/10/09 22:33:48 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:50 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:55 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_0, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.(ReduceTask.java:327)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:56 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:33:59 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:03 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_1, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.(ReduceTask.java:327)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:05 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000000_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:08 INFO mapred.JobClient: Task Id : task_200810092201_0010_m_000001_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:185)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:10 INFO mapred.JobClient: Task Id : task_200810092201_0010_r_000000_2, Status : FAILED
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:552)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.(ReduceTask.java:327)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 9 more
08/10/09 22:34:13 INFO mapred.JobClient: map 100% reduce 100%
08/10/09 22:34:14 INFO main.UpdateIndex: Elapsed time is 37s
Elapsed time is 37s
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
at org.apache.hadoop.contrib.index.mapred.IndexUpdater.run(IndexUpdater.java:53)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:263)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)

2008-10-10

chenlbspace

发件人： imcaptor
发送时间： 2008-10-10 17:07:39
收件人： [email protected]
抄送：
主题： Re: hadoop index org/apache/lucene/store/Directory NoClassDefFoundError

Try do this.
In the hadoop path,
-bash-3.00$ pwd
/data/hadoop/hadoop-0.18.1
cp src/contrib/index/lib/lucene-core-2.3.1.jar lib
Then you can run the task.
chenlbspace 写道:

__________ NOD32 3510 (20081010) Information __________
This message was checked by NOD32 antivirus system.
http://www.nod32cn.com

reply | permalink

Kyle Lau

at Oct 16, 2008 at 1:51 am

⇧

I ran into the same problem today. Found a solution that works:

bin/hadoop jar -libjars <comma separated jars> hadoop-xxx-index.jar
-inputPaths...

E.g., in your particular case, you command will look like:

bin/hadoop jar -libjars lib/lucene-core-2.3.1.jar
contrib/index/hadoop-0.17.1-index.jar -inputPaths
src/contrib/index/sample -outputPath index-msg-out -indexPath index
-conf src/contrib/index/conf/index-config.xml

Kyle

On Fri, 2008-10-10 at 16:35 +0800, chenlbspace wrote:
hi,

I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.

How solve? Thinks.

Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"

---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more

-----------------------------------------------------------------------------------------------------------------

2008-10-10

chenlbspace

reply | permalink

Chenlbspace

at Oct 16, 2008 at 5:44 am

⇧

Thinks, imcaptor and Kyle.

I add lucene-core-2.3.1.jar to lib, It can't correct run, but, suddenly no any error the day of tomorrow

2008-10-16

chenlbspace

发件人： chenlbspace
发送时间： 2008-10-10 16:35:40
收件人： [email protected]
抄送：
主题： hadoop index org/apache/lucene/store/Directory NoClassDefFoundError

hi,
I try to use contrib/index/hadoop-0.17.1-index.jar build lucene index, but Directory NoClassDefFoundError.
How solve? Thinks.
Additional, hadoop run Pseudo-Distributed Mode. and can run "bin/hadoop jar hadoop-*-examples.jar grep input output 'dfs[a-z.]+'"
---------------------------error----------------------------------------------------------------------------------
[[email protected] hadoop]$ bin/hadoop jar contrib/index/hadoop-0.17.1-index.jar -inputPaths src/contrib/index/sample -outputPath index-msg-out -indexPath index -conf src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: inputPaths = src/contrib/index/sample
08/10/09 21:41:28 INFO main.UpdateIndex: outputPath = index-msg-out
08/10/09 21:41:28 INFO main.UpdateIndex: shards = null
08/10/09 21:41:28 INFO main.UpdateIndex: indexPath = index
08/10/09 21:41:28 INFO main.UpdateIndex: numShards = -1
08/10/09 21:41:28 INFO main.UpdateIndex: numMapTasks= -1
08/10/09 21:41:28 INFO main.UpdateIndex: confPath = src/contrib/index/conf/index-config.xml
08/10/09 21:41:28 INFO main.UpdateIndex: Elapsed time is 0s
Elapsed time is 0s
java.lang.NoClassDefFoundError: org/apache/lucene/store/Directory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:581)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:599)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:625)
at org.apache.hadoop.contrib.index.mapred.IndexUpdateConfiguration.getIndexUpdaterClass(IndexUpdateConfiguration.java:145)
at org.apache.hadoop.contrib.index.main.UpdateIndex.main(UpdateIndex.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.Directory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more
-----------------------------------------------------------------------------------------------------------------
2008-10-10
chenlbspace
__________ NOD32 3510 (20081010) Information __________
This message was checked by NOD32 antivirus system.
http://www.nod32cn.com

reply | permalink

Related Discussions

Discussion Navigation

view	thread \| post

Discussion Overview

group	common-user @
categories	hadoop
posted	Oct 10, '08 at 8:36a
active	Oct 16, '08 at 5:44a
posts	7
users	3
website	hadoop.apache.org...
irc	#hadoop

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[Hadoop-common-user] hadoop index org/apache/lucene/store/Directory NoClassDefFoundError

Search Discussions

Related Discussions

3 users in discussion