Grokbase Groups Hive user July 2011
FAQ
Hi all,
I tried to index the lzo file but got the following error while indexing the
lzo file :

java.lang.ClassCastException:
com.hadoop.compression.lzo.LzopCodec$LzopDecompressor cannot be cast to
com.hadoop.compression.lzo.LzopDecompressor
at
com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)


I have performed following steps:-

1. $sudo apt-get install liblzo2-dev
2. Download the Hadoop-lzo clone from github repo (
https://github.com/kevinweil/hadoop-lzo )
3. Build the Hadoop-lzo project.
4. Copy the hadoop-lzo-*.jar file at $HADOOP_HOME/lib dir of cluster nodes
5. Copy the hadoop-lzo-install-dir/build/hadoop-lzo-*/native library at
$HADOOP_HOME/lib dir of cluster nodes.
6. core-site.xml:

<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec
,org.apache.hadoop.io.compress.DefaultCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,
org.apache.hadoop.io.compress.BZip2Codec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

7. mapred-site.xml:

<property>
<name>mapred.child.env</name>

<value>JAVA_LIBRARY_PATH=/opt/ladap/common/hadoop-0.20.2/lib/native/Linux-i386-32/*</value>

</property>

<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

8. hadoop-env.sh

export
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/ankit/hadoop-0.20.1/lib/hadoop-lzo-0.4.12.jar
export JAVA_LIBRARY_PATH=/home/ankit/hadoop-0.20.1/lib/native/Linux-i386-32/

9. Restart the cluster

10. uploaded lzo file into hdfs

11. Runned the following command for indexing:

bin/hadoop jar path/to/hadoop-lzo-*.jar
com.hadoop.compression.lzo.LzoIndexer lzofile.lzo

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJul 28, '11 at 5:28a
activeJul 28, '11 at 5:28a
posts1
users1
websitehive.apache.org

1 user in discussion

Ankit Jain: 1 post

People

Translate

site design / logo © 2021 Grokbase