FAQ
Hi all,
Thanks for sharing PigMix with us.
I do have a related question to this thread. The generate_data.sh references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar. This jar is never generated for me. I generated one out of the test class files and hope that it was it.
My other question, I tried to run the script but I keep getting this error:
Exception in thread "main" java.io.IOException: Error opening job jar: -libjars
at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(JarFile.java:133)
at java.util.jar.JarFile.(RunJar.java:88)

I made sure that the sdsuLibJKD12.jar is in place and I was able to compile the tests using it, but still the script will not work! Any suggestions.
Thanks




________________________________
From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
To: pig-user@hadoop.apache.org
Sent: Mon, November 2, 2009 12:01:00 PM
Subject: Re: DataGenerator Location

I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
cannot find any such classpath. Where can I expect to find this
DataGenerator class?
As Alan wrote earlier DataGenerator is in patch
https://issues.apache.org/jira/browse/PIG-200 You need to download the patch
and build it.
Finally, that Wiki page refers to specifying the path to $zipfjar:
sdsuLibJKD12.jar .  Where can I find this jar file?
http://www.eli.sdsu.edu/java-SDSU/

Hope it helps,
Ashutosh

Search Discussions

  • Rob Stewart at Dec 11, 2009 at 7:14 pm
    This is a good point.. the DataGenerator on the Wiki page needs updating.

    In Hadoop 0.20 the -libjars comes *after* the jar file.

    It should say:
    hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
    -libjars
    $zipfjar -conf $conf_file [options] colspec...


    Iman - Could you try doing this and let us know?

    Thanks,

    Rob Stewart


    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Hi all,
    Thanks for sharing PigMix with us.
    I do have a related question to this thread. The generate_data.sh
    references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
    This jar is never generated for me. I generated one out of the test class
    files and hope that it was it.
    My other question, I tried to run the script but I keep getting this error:
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

    I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
    the tests using it, but still the script will not work! Any suggestions.
    Thanks




    ________________________________
    From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
    To: pig-user@hadoop.apache.org
    Sent: Mon, November 2, 2009 12:01:00 PM
    Subject: Re: DataGenerator Location

    I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
    cannot find any such classpath. Where can I expect to find this
    DataGenerator class?
    As Alan wrote earlier DataGenerator is in patch
    https://issues.apache.org/jira/browse/PIG-200 You need to download the
    patch
    and build it.
    Finally, that Wiki page refers to specifying the path to $zipfjar:
    sdsuLibJKD12.jar . Where can I find this jar file?
    http://www.eli.sdsu.edu/java-SDSU/

    Hope it helps,
    Ashutosh


  • Iman E at Dec 11, 2009 at 10:08 pm
    Thankyou so much, Rob for your reply.

    Moving the -libjars fixed the error, but I still get this error:
    Exception in thread "main" java.lang.ClassNotFoundException: org.apache.pig.test.utils.datagen.DataGenerator
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

    The datagenjar (which is the test.jar that I have generated) has the org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix the error.
    Thanks
    Iman




    ________________________________
    From: Rob Stewart <robstewart57@googlemail.com>
    To: pig-user@hadoop.apache.org
    Sent: Fri, December 11, 2009 2:13:17 PM
    Subject: Re: DataGenerator Location

    This is a good point.. the DataGenerator on the Wiki page needs updating.

    In Hadoop 0.20 the -libjars comes *after* the jar file.

    It should say:
    hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
    -libjars
    $zipfjar -conf $conf_file [options] colspec...


    Iman - Could you try doing this and let us know?

    Thanks,

    Rob Stewart


    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Hi all,
    Thanks for sharing PigMix with us.
    I do have a related question to this thread. The generate_data.sh
    references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
    This jar is never generated for me. I generated one out of the test class
    files and hope that it was it.
    My other question, I tried to run the script but I keep getting this error:
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

    I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
    the tests using it, but still the script will not work! Any suggestions.
    Thanks




    ________________________________
    From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
    To: pig-user@hadoop.apache.org
    Sent: Mon, November 2, 2009 12:01:00 PM
    Subject: Re: DataGenerator Location

    I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
    cannot find any such classpath. Where can I expect to find this
    DataGenerator class?
    As Alan wrote earlier DataGenerator is in patch
    https://issues.apache.org/jira/browse/PIG-200 You need to download the
    patch
    and build it.
    Finally, that Wiki page refers to specifying the path to $zipfjar:
    sdsuLibJKD12.jar .  Where can I find this jar file?
    http://www.eli.sdsu.edu/java-SDSU/

    Hope it helps,
    Ashutosh


  • Rob Stewart at Dec 12, 2009 at 12:25 am
    OK, it seems you have an issue with your classpaths. It may be an idea to
    make a file called "generateData". Then in this, contain the following
    lines:
    -------------------------
    export pigjar=$PIGHOME/pig-0.4.0-core.jar
    export zipfjar=[LOCATION_TO]/sdsuLibJKD14.jar
    export datagenjar=[LOCATION_TO]/test.jar #(that you have created
    including the patch)
    export conf_file=$HADOOP_HOME/conf/hadoop-site.xml
    export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar
    $HADOOP_HOME/bin/hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -rows 100
    -f generated_data.dat s:8:50:z:0
    ----------------------

    Providing "echo $HADOOP_CLASSPATH" looks OK, there should be a file
    generated_data.dat in the same directory as the generateData file, once the
    generateData file has been executed.

    Let us know how you get on.


    Rob Stewart



    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Thankyou so much, Rob for your reply.

    Moving the -libjars fixed the error, but I still get this error:
    Exception in thread "main" java.lang.ClassNotFoundException:
    org.apache.pig.test.utils.datagen.DataGenerator
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

    The datagenjar (which is the test.jar that I have generated) has the
    org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix
    the error.
    Thanks
    Iman




    ________________________________
    From: Rob Stewart <robstewart57@googlemail.com>
    To: pig-user@hadoop.apache.org
    Sent: Fri, December 11, 2009 2:13:17 PM
    Subject: Re: DataGenerator Location

    This is a good point.. the DataGenerator on the Wiki page needs updating.

    In Hadoop 0.20 the -libjars comes *after* the jar file.

    It should say:
    hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
    -libjars
    $zipfjar -conf $conf_file [options] colspec...


    Iman - Could you try doing this and let us know?

    Thanks,

    Rob Stewart


    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Hi all,
    Thanks for sharing PigMix with us.
    I do have a related question to this thread. The generate_data.sh
    references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
    This jar is never generated for me. I generated one out of the test class
    files and hope that it was it.
    My other question, I tried to run the script but I keep getting this error:
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util.zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

    I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
    the tests using it, but still the script will not work! Any suggestions.
    Thanks




    ________________________________
    From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
    To: pig-user@hadoop.apache.org
    Sent: Mon, November 2, 2009 12:01:00 PM
    Subject: Re: DataGenerator Location

    I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
    cannot find any such classpath. Where can I expect to find this
    DataGenerator class?
    As Alan wrote earlier DataGenerator is in patch
    https://issues.apache.org/jira/browse/PIG-200 You need to download the
    patch
    and build it.
    Finally, that Wiki page refers to specifying the path to $zipfjar:
    sdsuLibJKD12.jar . Where can I find this jar file?
    http://www.eli.sdsu.edu/java-SDSU/

    Hope it helps,
    Ashutosh




  • Iman E at Dec 15, 2009 at 5:38 am
    Thanks a lot Rob. It worked. It turned to be that I had a problem with my test.jar.





    ________________________________
    From: Rob Stewart <robstewart57@googlemail.com>
    To: pig-user@hadoop.apache.org
    Sent: Fri, December 11, 2009 7:25:00 PM
    Subject: Re: DataGenerator Location

    OK, it seems you have an issue with your classpaths. It may be an idea to
    make a file called "generateData". Then in this, contain the following
    lines:
    -------------------------
    export pigjar=$PIGHOME/pig-0.4.0-core.jar
    export zipfjar=[LOCATION_TO]/sdsuLibJKD14.jar
    export datagenjar=[LOCATION_TO]/test.jar    #(that you have created
    including the patch)
    export conf_file=$HADOOP_HOME/conf/hadoop-site.xml
    export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar
    $HADOOP_HOME/bin/hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -rows 100
    -f generated_data.dat s:8:50:z:0
    ----------------------

    Providing "echo $HADOOP_CLASSPATH" looks OK, there should be a file
    generated_data.dat in the same directory as the generateData file, once the
    generateData file has been executed.

    Let us know how you get on.


    Rob Stewart



    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Thankyou so much, Rob for your reply.

    Moving the -libjars fixed the error, but I still get this error:
    Exception in thread "main" java.lang.ClassNotFoundException:
    org.apache.pig.test.utils.datagen.DataGenerator
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader..loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

    The datagenjar (which is the test.jar that I have generated) has the
    org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix
    the error.
    Thanks
    Iman




    ________________________________
    From: Rob Stewart <robstewart57@googlemail.com>
    To: pig-user@hadoop.apache.org
    Sent: Fri, December 11, 2009 2:13:17 PM
    Subject: Re: DataGenerator Location

    This is a good point... the DataGenerator on the Wiki page needs updating.

    In Hadoop 0.20 the -libjars comes *after* the jar file.

    It should say:
    hadoop jar $datagenjar
    org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator>
    -libjars
    $zipfjar -conf $conf_file [options] colspec....


    Iman - Could you try doing this and let us know?

    Thanks,

    Rob Stewart


    2009/12/11 Iman E <hadoop_ami@yahoo.com>
    Hi all,
    Thanks for sharing PigMix with us.
    I do have a related question to this thread. The generate_data.sh
    references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar.
    This jar is never generated for me. I generated one out of the test class
    files and hope that it was it.
    My other question, I tried to run the script but I keep getting this error:
    Exception in thread "main" java.io.IOException: Error opening job jar:
    -libjars
    at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
    Caused by: java.util..zip.ZipException: error in opening zip file
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:114)
    at java.util.jar.JarFile.<init>(JarFile.java:133)
    at java.util.jar.JarFile.<init>(JarFile.java:70)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:88)

    I made sure that the sdsuLibJKD12.jar is in place and I was able to compile
    the tests using it, but still the script will not work! Any suggestions.
    Thanks




    ________________________________
    From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
    To: pig-user@hadoop.apache..org
    Sent: Mon, November 2, 2009 12:01:00 PM
    Subject: Re: DataGenerator Location

    I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and
    cannot find any such classpath. Where can I expect to find this
    DataGenerator class?
    As Alan wrote earlier DataGenerator is in patch
    https://issues.apache.org/jira/browse/PIG-200 You need to download the
    patch
    and build it.
    Finally, that Wiki page refers to specifying the path to $zipfjar:
    sdsuLibJKD12.jar .  Where can I find this jar file?
    http://www.eli.sdsu.edu/java-SDSU/

    Hope it helps,
    Ashutosh




Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedDec 11, '09 at 6:57p
activeDec 15, '09 at 5:38a
posts5
users2
websitepig.apache.org

2 users in discussion

Iman E: 3 posts Rob Stewart: 2 posts

People

Translate

site design / logo © 2021 Grokbase