We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
node, which is on EC2 LS , which has a few jar files. When we start hive, it
fails to start
Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
HIVE-487 patch
It looks like the invocation command line is changed/different. Or maybe we
have a setup issue? Any idea why this is happening?
Thanks in advance,
Eva.
Here is the command line that starts hive (extracted from ps):
/usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Dhadoop.policy.file=hadoop-policy.xml -classpath
/etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/
usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/l
ib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.j
ar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/comm
ons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:
/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/
lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/h
adoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.
20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.j
ar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/l
ib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoo
p-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/u
sr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.j
ar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hado
op-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.ja
r:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4
j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop
-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.j
ar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/
dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/an
tlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.ja
r:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/
usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-
r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r78664
8/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/
build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/
hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648
/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/..
/lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_ser
de.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/b
uild/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/.
./lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-e
nhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2
.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r7
86648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist
/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthri
ft.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r
786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/Pars
eNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveud
f.jar org.apache.hadoop.util.RunJar -libjars
file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dis
t/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///hom
e/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
Error we get:
[dataeng@master nfdev]$ hive
Exception in thread "main" java.io.IOException: Error opening job jar:
-libjars
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(JarFile.java:133)
at java.util.jar.JarFile.(RunJar.java:88)
On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
Hive) on a different cluster. Here is the last portion of the command line
when starting hive and it would work:
:/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/ets
e/nfdev/dist/ParseNCCP.jar org.apache.hadoop.mapred.JobShell -libjars
file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.
jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
/apps/hadoop/hive/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/
nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
