Grokbase Groups Pig user August 2009
FAQ
Yes, I use PIG-660_4.patch. And I check out from svn.

I will try again. Thank you.

George

2009/9/4 Turner Kunkel <thkunkel@gmail.com>
Did you patch using PIG-660_4.patch?

It looks like it's not looking in the correct spot for Java classes.
Did you download everything from the svn Pig package correctly?

I also ran into the issue of running ant with certain permissions. I
dedicated full rights to user 'hadoop' on all of my Hadoop folders
(pig, hadoop, etc.) and ant only succeeded when I was logged in as
user 'hadoop' and ran ant.

-Turner

On Fri, Sep 4, 2009 at 3:15 PM, George Pangwrote:
Hi Turner,

I use Alex's tutorial on building Pig with Ant but get errors like this,

........
er.java:36: package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.LogFactory;
[javac] ^
[javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:40:
package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.Log;
[javac] ^
[javac] /home/hadoop20/pig/src/org/apache/pig/impl/PigContext.java:41:
package org.apache.commons.logging does not exist
[javac] import org.apache.commons.logging.LogFactory;
[javac] ^
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors

BUILD FAILED
/home/hadoop20/pig/build.xml:246: The following error occurred while
executing this line:
/home/hadoop20/pig/build.xml:285: Compile failed; see the compiler error
output for details.

Where did I get wrong? Or where to find the error output file?

Thank you.

George



2009/8/20 Turner Kunkel <thkunkel@gmail.com>
I built hadoop20.jar using ant after patching the Hadoop 0.20.0
download with the patch provided on Alex's how-to site.
I grabbed Pig using svn and patched it with PIG-660_4.patch, then
moved hadoop20.jar into Pig's lib directory, then built pig.jar with
ant.
I set my classpath to include the hadoop conf folder and pig.jar
(along with the other environment settings).
I execute pig with this command:

java -Dfs.default.name=hdfs://localhost:9000
-Dmapred.job.tracker=localhost:9001 -jar pig.jar (with my own hdfs
and mapred settings of course)

So I was doing multiple things differently after coming from Hadoop
18, but this seems to work.

-Turner

On Thu, Aug 20, 2009 at 11:38 AM, Dmitriy Ryaboy <dvryaboy@cloudera.com
wrote:
Turner,
Before you went through those steps, did you use a vanilla hadoop 20
jar or did you get one from the zebra ticket?
I am still at a loss as to why it wasn't working for you before.
-D

On Thu, Aug 20, 2009 at 6:29 AM, Turner Kunkel<thkunkel@gmail.com>
wrote:
<http://behemoth.strlen.net/%7Ealex/hadoop20-pig-howto.txt>,
I got it to work.
Thanks much, this helps me a lot. Have a nice day.

-Turner

On Wed, Aug 19, 2009 at 4:44 PM, Dmitriy Ryaboy <
dvryaboy@cloudera.com
wrote:
Tumer,
That error means you dropped pig.jar from your classpath.

On Wed, Aug 19, 2009 at 12:50 PM, Turner Kunkel<thkunkel@gmail.com
wrote:
Ok, I know I'm really close now, I ran into this when running Pig
on
18.
"Could not find the main class: org.apache.pig.Main"

I forgot how to resolve this.

Thanks a lot so far, by the way :)

-Turner

On Wed, Aug 19, 2009 at 2:06 PM, Santhosh Srinivasan <
sms@yahoo-inc.com
wrote:
AFAIK, hadoop-20 does not support hod. As Chad suggested drop
the
hod
parameters and thing should start working

-----Original Message-----
From: Naber, Chad
Sent: Wednesday, August 19, 2009 11:52 AM
To: pig-user@hadoop.apache.org
Subject: RE: Pig 0.3.0 and Hadoop 0.20.0

If you are not using Hadoop on Demand (hod), you will need to
comment
out the hod properties in pig's conf dir.

-----Original Message-----
From: Turner Kunkel
Sent: Wednesday, August 19, 2009 11:50 AM
To: pig-user@hadoop.apache.org
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Getting closer...
Actually got and error this time:
ERROR 6012: Unable to run command: hod allocate ....

-Turner

On Wed, Aug 19, 2009 at 12:24 PM, Santhosh Srinivasan
wrote:
You need a special hadoop20.jar to build the pig sources.
Check
out
PIG-660
Santhosh


-----Original Message-----
From: Turner Kunkel
Sent: Wed 8/19/2009 9:53 AM
To: pig-user@hadoop.apache.org
Subject: Re: Pig 0.3.0 and Hadoop 0.20.0

Hm, still nothing. Maybe I have to build it differently? I
will
play
around with the environment settings, but any more input is
appreciated.
-Turner

On Wed, Aug 19, 2009 at 10:09 AM, Dmitriy Ryaboy
<dvryaboy@cloudera.com
wrote:
Don't point it at mapred-site.xml directly, but the conf
directory,
instead.
(it is smart enough to pick up the new config files when
it's
running on
20).

I tried setting the path directly to mapred-site.xml like
you
have
it,
and indeed, that doesn't work. Changing to the directory
should
fix
your issue.

-D

On Wed, Aug 19, 2009 at 6:26 AM, Turner Kunkel<
thkunkel@gmail.com>
wrote:
I am running off of unpacked tar.gz files of Hadoop and
Pig.
Was
working
with 0.18.0 and had it working fine.
With 0.20.0 I imagine you're correct with changing
PIG_CLASSPATH
since
Pig
is looking for hadoop-site.xml instead of mapred-site.xml.
My environment looks like this now:

JAVA_HOME=/usr/lib/jvm/java-6-sun
PIG_CLASSPATH=/usr/local/pig-0.3.0/pig-0.3.0-core.jar:/usr/local/hadoop/
conf/mapred-site.xml
PIGDIR=/usr/local/pig-0.3.0
HADOOP_HOME=/usr/local/hadoop
HADOOPDIR=/usr/local/hadoop/conf
PIG_HADOOP_VERSION=20

Pig still isn't connecting correctly.

-Turner

On Tue, Aug 18, 2009 at 4:59 PM, Dmitriy Ryaboy
<dvryaboy@cloudera.com
wrote:
Nipun and Turner,
What are you setting PIG_CLASSPATH to?

My environment works if I set it to
/path/to/pig.jar:path/to/mapred-site.xml

(leaving off the path to mapred-site.xml or pig.jar both
lead to
breakage -- I haven't quite decided if that's a bug or
not.)
For completeness, a full set of environment variables
that
make
it
work for me. I am running the Cloudera CDH2 testing
package
for
debian/ubuntu (from http://archive.cloudera.com) and a
patched
version
of Pig from trunk.

JAVA_HOME=/usr/lib/jvm/jdk1.6.0_14/
PIGDIR=/home/dvryaboy/src/pig/trunk/
PIG_HADOOP_VERSION=20
PIG_CONF_DIR=./conf/
PIG_CLASSPATH=/etc/hadoop-0.20/conf.pseudo/:/home/dvryaboy/src/pig/trunk
/pig.jar

On Tue, Aug 18, 2009 at 10:37 AM, Turner
Kunkel<thkunkel@gmail.com>
wrote:
Hello,
I upgraded to Hadoop 0.20.0 (it's stable and working)
and
Pig
no
longer
connects to the HDFS.
I tried rebuilding and applying patch PIG660.
I have a script that I run that exports these
environment
tables
correctly:
JAVA_HOME, PIGDIR, PIG_CLASSPATH, HADOOP_HOME
Pig still connects to file:/// instead of HDFS.
I rebuilt the Pig package so that there are no patches
applied
at
the
moment, as if I had just unpacked PIG-0.3.0 for the
first
time.
So, what do I do to get Pig 0.3.0 to work with Hadoop
0.20.0
from
the
start?
Thanks!

--

-Turner Kunkel

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 17 of 17 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedAug 18, '09 at 5:37p
activeSep 5, '09 at 12:54a
posts17
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase