Grokbase Groups Pig user May 2008
FAQ
What's currently on top of trunk requires use of hadoop17.jar instead of
hadoop16.jar.

Alan.

Iván de Prado wrote:
I did ant clean, and tried to recompile with hadoop16.jar. But I got
these compilation errors:

[javac] Compiling 241 source files to /opt/hd1/pig-trunk/build/classes
[javac] /opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:28: cannot find symbol
[javac] symbol : class FileOutputFormat
[javac] location: package org.apache.hadoop.mapred
[javac] import org.apache.hadoop.mapred.FileOutputFormat;
[javac] ^
[javac] /opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/SortPartitioner.java:26: cannot find symbol
[javac] symbol : class RawComparator
[javac] location: package org.apache.hadoop.io
[javac] import org.apache.hadoop.io.RawComparator;
[javac] ^
[javac] /opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/SortPartitioner.java:37: cannot find symbol
[javac] symbol : class RawComparator
[javac] location: class org.apache.pig.backend.hadoop.executionengine.mapreduceExec.SortPartitioner
[javac] RawComparator comparator;
[javac] ^
[javac] /opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:48: cannot find symbol
[javac] symbol : variable FileOutputFormat
[javac] location: class org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat
[javac] Path outputDir = FileOutputFormat.getWorkOutputPath(job);
[javac] ^
[javac] /opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:62: cannot find symbol
[javac] symbol : variable FileOutputFormat
[javac] location: class org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat
[javac] String parentName = FileOutputFormat.getOutputPath(job).getName();


Is not Pig compatible with Hadoop 0.16 anymore? Did I do something wrong when compiling?

I'm using the revision 661633

Iván

El vie, 30-05-2008 a las 13:06 +0200, Iván de Prado escribió:
With the latest version I'm getting an error:

java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/FileOutputFormat
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat.getRecordWriter(PigOutputFormat.java:48)
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:257)
at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:111)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.FileOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 5 more

What did I do wrong? I'm launching pig using bin/pig script. Before the update, it worked.

Iván de Prado
www.ivanprado.es


El vie, 30-05-2008 a las 20:01 +1000, pi song escribió:
We've already fixed the memory issue introduced in Pig-85. Could you please
update to the latest version and try again?

Pi

On Wed, May 28, 2008 at 9:18 AM, pi song wrote:

This might have nothing to do with Hadoop 0.17 but something else that we
fixed right after it. I'm investigating. Sorry for inconvenience.

FYI,
Pi

On 5/28/08, Tanton Gibbs wrote:

I think you need to increase the amount of memory you give to java.

It looks like it is currently set to 256M. I upped mine to 2G. Of
course it depends on how much ram you have available.

mapred.child.java.opts is the parameter
mine is currently set to 2048M in my hadoop-site.xml file.

For performance reasons, I upped the io.sort.mb parameter. However,
if this is too close to 50% of the total memory, you will get the
Spillable messages.

HTH,
Tanton

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 11 of 12 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 23, '08 at 5:52a
activeMay 30, '08 at 4:32p
posts12
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase