[Pig-user] ERROR 2999: Unexpected internal error. Failed to create DataStorage

Hello,

I have been trying to set up pig 0.8.1 to work with hadoop 0.20.203
without success. At the moment, if I run pig -x local I correctly get
access to the grunt shell but when I try to run the default mapreduce
option it fails with error 2999, see below:

miguel@euler:~$ pig
2011-05-23 19:35:09,843 [main] INFO org.apache.pig.Main - Logging
error messages to: /home/miguel/pig_1306168509838.log
2011-05-23 19:35:10,030 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://localhost:54310
2011-05-23 19:35:10,168 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Failed to create DataStorage
Details at logfile: /home/miguel/pig_1306168509838.log

Hadoop itself runs normally on my system. Any ideas on what to do to
fix this? Is it an incompatibility issue or something I am doing
wrong? Below I am including my system's information, hopefully this
will be useful for a diagnosis.

The log file has the following:

Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
at org.apache.pig.PigServer.<init>(PigServer.java:226)
at org.apache.pig.PigServer.<init>(PigServer.java:215)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
at org.apache.pig.Main.run(Main.java:452)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to localhost/127.0.0.1:54310
failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

================================================================================

My system details:

Ubuntu 10.04 LTS
miguel@euler:~$ pig -version
Apache Pig version 0.8.1 (r1094835)
compiled Apr 18 2011, 19:26:53
miguel@euler:~$ hadoop version
Hadoop 0.20.203.0
Subversion
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
-r 1099333
Compiled by oom on Wed May 4 07:57:50 PDT 2011

My environment variables (I know I have more than I need but tutorials
on the web have been a little hazy on this):

miguel@euler:~$ printenv | grep "PIG\|HADOOP"
PIG_DIR=/home/miguel/pig/pig-0.8.1
PIG_HOME=/home/miguel/pig/pig-0.8.1
HADOOP_HOME=/usr/local/hadoop
PIGDIR=/home/miguel/pig/pig-0.8.1
PIG_INSTALL=/home/miguel/pig/pig-0.8.1
PIG_HADOOP_VERSION=20
HADOOP_INSTALL=/usr/local/hadoop
PIG_CLASSPATH=/usr/local/hadoop/conf
HADOOPDIR=/usr/local/hadoop/conf

Many thanks for any help,
Rui Miguel Forte

On 23 May 2011 20:35, Jonathan Coveney wrote:
Here is what I had to do to get pig running with a different version of
Hadoop (in my case, the cloudera build but I'd try this as well):

build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
you run pig, put the pig-withouthadoop.jar on your classpath as well as your
hadoop jar. In my case, I found that scripts only worked if I additionally
manually registered the antlr jar:

register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;

2011/5/23 Rui Miguel Forte <ruimiguelforte@gmail.com>

Hi Jonathan,

Sorry or the late response, First of all thanks for your email. I'm
afraid I still havent managed to get this to work. I am not clear on
what you mean by register the antlr jar script though. How do I do
this?

Thanks,
Miguel


org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
================================================================================
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203