Grokbase Groups Pig user May 2009
FAQ
Hi,
I added only this, and it works now.

*% export PIG_HADOOP_VERSION=18*

George


2009/5/28 George Pang <p0941p@gmail.com>
Obviously my program has problem going to this line:

481 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: localhost:54311

I got this line from Cloudera's VM.

George


2009/5/27 George Pang <p0941p@gmail.com>
Correction: the port data seems strange. After I
restart Ubuntu, it is like:

wpang2@wpang2-desktop:~$ sudo netstat -plten | grep java
tcp6 0 0 127.0.0.1:48961 :::*
LISTEN 1000 19407 6067/java
tcp6 0 0 :::50020 :::*
LISTEN 1000 19428 5826/java
tcp6 0 0 127.0.0.1:54310 :::*
LISTEN 1000 18602 5745/java
tcp6 0 0 127.0.0.1:54311 :::*
LISTEN 1000 18979 5979/java
tcp6 0 0 :::59271 :::*
LISTEN 1000 18849 5911/java
tcp6 0 0 :::50090 :::*
LISTEN 1000 19421 5911/java
tcp6 0 0 :::49067 :::*
LISTEN 1000 18754 5826/java
tcp6 0 0 :::50060 :::*
LISTEN 1000 19402 6067/java
tcp6 0 0 :::50030 :::*
LISTEN 1000 19341 5979/java
tcp6 0 0 :::40047 :::*
LISTEN 1000 18932 5979/java
tcp6 0 0 :::50070 :::*
LISTEN 1000 19002 5745/java
tcp6 0 0 :::40887 :::*
LISTEN 1000 18507 5745/java
tcp6 0 0 :::50010 :::*
LISTEN 1000 19059 5826/java
tcp6 0 0 :::50075 :::*
LISTEN 1000 19422 5826/java

George

2009/5/27 George Pang <p0941p@gmail.com>

Hi Chad,
I think you maybe right. I run fsck, and get this:
fsck.ext3: Unable to resolve 'UUID=28038a1a-8172-41dd-9bb7-25882543b7cf'

Since I am a Windows-turned linux user, I don't know much about this
error. In an initial search online, I think it's something wrong with my
disk partition. However I run my Ubuntu on a Virtual Machine. I hope this
won't make things more complex.

Other than this, my hadoop is now a single-node cluster. Some statuss
are as follows:

wpang2@wpang2-desktop:~$* jps*
6039 NameNode
6103 DataNode
6180 SecondaryNameNode
6332 TaskTracker
6254 JobTracker
8139 Jps

wpang2@wpang2-desktop:~$ sudo *netstat -plten | grep java*
[sudo] password for wpang2:
tcp6 0 0 :::50020 :::*
LISTEN 1000 21585 6103/java
tcp6 0 0 127.0.0.1:59621 :::*
LISTEN 1000 21319 6332/java
tcp6 0 0 127.0.0.1:9000 :::*
LISTEN 1000 20964 6039/java
tcp6 0 0 127.0.0.1:9001 :::*
LISTEN 1000 20962 6254/java
tcp6 0 0 :::50090 :::*
LISTEN 1000 21567 6180/java
tcp6 0 0 :::54539 :::*
LISTEN 1000 20937 6103/java
tcp6 0 0 :::50060 :::*
LISTEN 1000 21313 6332/java
tcp6 0 0 :::48237 :::*
LISTEN 1000 20941 6180/java
tcp6 0 0 :::50030 :::*
LISTEN 1000 21315 6254/java
tcp6 0 0 :::46511 :::*
LISTEN 1000 20943 6039/java
tcp6 0 0 :::43189 :::*
LISTEN 1000 20939 6254/java
tcp6 0 0 :::50070 :::*
LISTEN 1000 21318 6039/java
tcp6 0 0 :::50010 :::*
LISTEN 1000 21485 6103/java
tcp6 0 0 :::50075 :::*
LISTEN 1000 21569 6103/java

Thank you,

George


2009/5/27 Naber, Chad <CNaber@edmunds.com>

George,

I would bet that there are some problems with the HDFS
'hdfs://localhost:54310' you are connecting to. Have you verified that the
HDFS is up and is healthy? Maybe do an FSCK on the blocks to verify that
there are no corrupt blocks.

Chad




-----Original Message-----
From: George Pang
Sent: Wednesday, May 27, 2009 12:40 PM
To: pig-user@hadoop.apache.org
Subject: Re: Pig running hadoop mode

Hi Alan,
I set these 3 environment variables,

wpang@wpang-desktop:~$ echo *$PIGDIR*
/home/wpang/pigtmp
wpang@wpang-desktop:~$ echo *$PIG_CLASSPATH*
/home/wpang/pigtmp/pig.jar:/usr/local/hadoop/hadoop-0.18.3/conf
wpang@wpang-desktop:~$ echo *$JAVA_HOME*
/usr/lib/jvm/jdk1.6.0_13

So the hadoop version is 0.18.3.

I used to download the pig.jar from "pig tutorial" wiki page and
pig-0.2.0 release from the release page. Now I try to build the pig from
subversion, and it gave me the same thing.

Any idea will be much appreciated.

George



2009/5/27 Alan Gates <gates@yahoo-inc.com>
What version of hadoop are you using? Pig, as currently released,
runs on 0.18.x hadoop. If you're running a different version you made
need to apply a patch to make it run with your version of hadoop.

Alan.


On May 26, 2009, at 8:39 PM, George Pang wrote:

Dear users,
Now my pig can connect to my hdfs, as on the screeen:
wpang2@wpang2-desktop:~/PigTrunk/trunk/bin$ pig
2009-05-26 20:22:29,243 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://localhost:54310
..........

But then it failed and generate an error log:
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:194)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133)
at org.apache.pig.impl.PigContext.connect(PigContext.java:175)
at org.apache.pig.PigServer.<init>(PigServer.java:144)
at org.apache.pig.PigServer.<init>(PigServer.java:132)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:51)
at org.apache.pig.Main.main(Main.java:318)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:559)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
at
org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 8 more

Your advice is much appreciated!
George

2009/5/25 George Pang <p0941p@gmail.com>

Hi users,
When I run pig in hadoop mode, before the grunt shell appears there
is something on the command prompt not quite right:
"Initializing JVM Metrics with processName=JobTracker, sessionId=
grunt> "

If I use grunt> ls , it only displays the local files, not on hdfs.
Could
anyone help with these?

George

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 13 of 13 | next ›
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 25, '09 at 6:43p
activeMay 29, '09 at 2:11a
posts13
users3
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase