Grokbase Groups Hive user July 2009
FAQ
We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
node, which is on EC2 LS , which has a few jar files. When we start hive, it
fails to start

Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
HIVE-487 patch

It looks like the invocation command line is changed/different. Or maybe we
have a setup issue? Any idea why this is happening?

Thanks in advance,
Eva.

Here is the command line that starts hive (extracted from ps):
/usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Dhadoop.policy.file=hadoop-policy.xml -classpath
/etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/
usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/l
ib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.j
ar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/comm
ons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:
/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/
lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/h
adoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.
20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.j
ar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/l
ib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoo
p-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/u
sr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.j
ar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hado
op-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.ja
r:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4
j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop
-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.j
ar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/
dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/an
tlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.ja
r:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/
usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-
r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r78664
8/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/
build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/
hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/us
r/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648
/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/..
/lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_ser
de.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/b
uild/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/.
./lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-e
nhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2
.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r7
86648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist
/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthri
ft.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib
/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r
786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/Pars
eNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveud
f.jar org.apache.hadoop.util.RunJar -libjars
file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dis
t/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///hom
e/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar

Error we get:
[dataeng@master nfdev]$ hive
Exception in thread "main" java.io.IOException: Error opening job jar:
-libjars
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(JarFile.java:133)
at java.util.jar.JarFile.(RunJar.java:88)


On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
Hive) on a different cluster. Here is the last portion of the command line
when starting hive and it would work:

:/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/ets
e/nfdev/dist/ParseNCCP.jar org.apache.hadoop.mapred.JobShell -libjars
file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.
jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
/apps/hadoop/hive/bin/../lib/hive_cli.jar
org.apache.hadoop.hive.cli.CliDriver -hiveconf
hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/
nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar

Search Discussions

  • Zheng Shao at Jul 9, 2009 at 12:50 am
    It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
    Can you try remove the "-libjars xxx" from the command line?
    It's added somewhere in bin/hive (or scripts that got called by bin/hive)

    Zheng

    On Wed, Jul 8, 2009 at 3:45 PM, Eva Tsewrote:
    We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
    node, which is on EC2 LS , which has a few jar files. When we start hive, it
    fails to start

    Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
    HIVE-487 patch

    It looks like the invocation command line is changed/different. Or maybe we
    have a setup issue? Any idea why this is happening?

    Thanks in advance,
    Eva.

    Here is the command line that starts hive (extracted from ps):
    /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
    -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
    -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
    -Dhadoop.policy.file=hadoop-policy.xml -classpath
    /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveudf.jar
    org.apache.hadoop.util.RunJar -libjars
    file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
    /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar

    Error we get:
    [dataeng@master nfdev]$ hive
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)


    On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
    Hive) on a different cluster. Here is the last portion of the command line
    when starting hive and it would work:

    :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse/nfdev/dist/ParseNCCP.jar
    org.apache.hadoop.mapred.JobShell -libjars
    file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
    /apps/hadoop/hive/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar


    --
    Yours,
    Zheng
  • Joydeep Sen Sarma at Jul 9, 2009 at 12:58 am
    Sorry - this is also needed as part of hive-487:

    In hadoop-20 - the -libjars has to come after the jar file/class

    Please try applying this patch to bin/ext/cli.sh

    --- cli.sh (revision 789726)
    +++ cli.sh (working copy)
    @@ -10,7 +10,7 @@
    exit 3;
    fi

    - exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS $HIVE_OPTS "$@"
    + exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE $HIVE_OPTS "$@"
    }


    I will add this to the 487 patch.

    -----Original Message-----
    From: Zheng Shao
    Sent: Wednesday, July 08, 2009 5:51 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...

    It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
    Can you try remove the "-libjars xxx" from the command line?
    It's added somewhere in bin/hive (or scripts that got called by bin/hive)

    Zheng

    On Wed, Jul 8, 2009 at 3:45 PM, Eva Tsewrote:
    We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
    node, which is on EC2 LS , which has a few jar files. When we start hive, it
    fails to start

    Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
    HIVE-487 patch

    It looks like the invocation command line is changed/different. Or maybe we
    have a setup issue? Any idea why this is happening?

    Thanks in advance,
    Eva.

    Here is the command line that starts hive (extracted from ps):
    /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
    -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
    -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
    -Dhadoop.policy.file=hadoop-policy.xml -classpath
    /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveudf.jar
    org.apache.hadoop.util.RunJar -libjars
    file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
    /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar

    Error we get:
    [dataeng@master nfdev]$ hive
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)


    On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
    Hive) on a different cluster. Here is the last portion of the command line
    when starting hive and it would work:

    :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse/nfdev/dist/ParseNCCP.jar
    org.apache.hadoop.mapred.JobShell -libjars
    file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
    /apps/hadoop/hive/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar


    --
    Yours,
    Zheng
  • Eva Tse at Jul 9, 2009 at 5:34 pm
    Thanks Joydeep & Zheng for the feedback.
    We will apply the new 487 patch when it is ready.

    Thanks!
    Eva.

    On 7/8/09 5:58 PM, "Joydeep Sen Sarma" wrote:

    Sorry - this is also needed as part of hive-487:

    In hadoop-20 - the -libjars has to come after the jar file/class

    Please try applying this patch to bin/ext/cli.sh

    --- cli.sh (revision 789726)
    +++ cli.sh (working copy)
    @@ -10,7 +10,7 @@
    exit 3;
    fi

    - exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS
    $HIVE_OPTS "$@"
    + exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE
    $HIVE_OPTS "$@"
    }


    I will add this to the 487 patch.

    -----Original Message-----
    From: Zheng Shao
    Sent: Wednesday, July 08, 2009 5:51 PM
    To: hive-user@hadoop.apache.org
    Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...

    It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
    Can you try remove the "-libjars xxx" from the command line?
    It's added somewhere in bin/hive (or scripts that got called by bin/hive)

    Zheng

    On Wed, Jul 8, 2009 at 3:45 PM, Eva Tsewrote:
    We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
    node, which is on EC2 LS , which has a few jar files. When we start hive, it
    fails to start

    Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
    HIVE-487 patch

    It looks like the invocation command line is changed/different. Or maybe we
    have a setup issue? Any idea why this is happening?

    Thanks in advance,
    Eva.

    Here is the command line that starts hive (extracted from ps):
    /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
    -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
    -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
    -Dhadoop.policy.file=hadoop-policy.xml -classpath
    /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/u
    sr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib
    /commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:
    /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-
    httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/
    lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/co
    mmons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0
    .20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/h
    sqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/l
    ib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-
    0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/
    jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoo
    p-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/h
    adoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/or
    o-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hado
    op-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.
    jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1
    /jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r
    786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/Tes
    tSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.ja
    r:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r7866
    48/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/b
    uild/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/buil
    d/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/.
    ./lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/c
    ommons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derb
    y.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-
    r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dis
    t/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_j
    dbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/l
    ib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/bu
    ild/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../
    lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94
    .jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib
    /hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r78
    6648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/d
    ist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8
    .1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive
    -r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist
    /bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stri
    ngtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5
    .jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.j
    ar:/home/dataeng/nfdev/dist/hiveudf.jar
    org.apache.hadoop.util.RunJar -libjars
    file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist
    /geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
    /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home
    /dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar

    Error we get:
    [dataeng@master nfdev]$ hive
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)


    On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
    Hive) on a different cluster. Here is the last portion of the command line
    when starting hive and it would work:

    :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse
    /nfdev/dist/ParseNCCP.jar
    org.apache.hadoop.mapred.JobShell -libjars
    file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.j
    ar,file:///home/etse/nfdev/dist/ParseNCCP.jar
    /apps/hadoop/hive/bin/../lib/hive_cli.jar
    org.apache.hadoop.hive.cli.CliDriver -hiveconf
    hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/n
    fdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedJul 8, '09 at 10:45p
activeJul 9, '09 at 5:34p
posts4
users3
websitehive.apache.org

People

Translate

site design / logo © 2022 Grokbase