Grokbase Groups Pig user May 2011
FAQ
Hello,

I have been trying to set up pig 0.8.1 to work with hadoop 0.20.203
without success. At the moment, if I run pig -x local I correctly get
access to the grunt shell but when I try to run the default mapreduce
option it fails with error 2999, see below:

miguel@euler:~$ pig
2011-05-23 19:35:09,843 [main] INFO org.apache.pig.Main - Logging
error messages to: /home/miguel/pig_1306168509838.log
2011-05-23 19:35:10,030 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://localhost:54310
2011-05-23 19:35:10,168 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Failed to create DataStorage
Details at logfile: /home/miguel/pig_1306168509838.log

Hadoop itself runs normally on my system. Any ideas on what to do to
fix this? Is it an incompatibility issue or something I am doing
wrong? Below I am including my system's information, hopefully this
will be useful for a diagnosis.

The log file has the following:

Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:214)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
at org.apache.pig.PigServer.(PigServer.java:215)
at org.apache.pig.tools.grunt.Grunt.(Main.java:452)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to localhost/127.0.0.1:54310
failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
================================================================================

My system details:

Ubuntu 10.04 LTS
miguel@euler:~$ pig -version
Apache Pig version 0.8.1 (r1094835)
compiled Apr 18 2011, 19:26:53
miguel@euler:~$ hadoop version
Hadoop 0.20.203.0
Subversion http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
-r 1099333
Compiled by oom on Wed May 4 07:57:50 PDT 2011

My environment variables (I know I have more than I need but tutorials
on the web have been a little hazy on this):

miguel@euler:~$ printenv | grep "PIG\|HADOOP"
PIG_DIR=/home/miguel/pig/pig-0.8.1
PIG_HOME=/home/miguel/pig/pig-0.8.1
HADOOP_HOME=/usr/local/hadoop
PIGDIR=/home/miguel/pig/pig-0.8.1
PIG_INSTALL=/home/miguel/pig/pig-0.8.1
PIG_HADOOP_VERSION=20
HADOOP_INSTALL=/usr/local/hadoop
PIG_CLASSPATH=/usr/local/hadoop/conf
HADOOPDIR=/usr/local/hadoop/conf

Many thanks for any help,
Rui Miguel Forte

Search Discussions

  • Jonathan Coveney at May 23, 2011 at 5:36 pm
    Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):

    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:

    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;

    2011/5/23 Rui Miguel Forte <ruimiguelforte@gmail.com>
    Hello,

    I have been trying to set up pig 0.8.1 to work with hadoop 0.20.203
    without success. At the moment, if I run pig -x local I correctly get
    access to the grunt shell but when I try to run the default mapreduce
    option it fails with error 2999, see below:

    miguel@euler:~$ pig
    2011-05-23 19:35:09,843 [main] INFO org.apache.pig.Main - Logging
    error messages to: /home/miguel/pig_1306168509838.log
    2011-05-23 19:35:10,030 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting to hadoop file system at: hdfs://localhost:54310
    2011-05-23 19:35:10,168 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage
    Details at logfile: /home/miguel/pig_1306168509838.log

    Hadoop itself runs normally on my system. Any ideas on what to do to
    fix this? Is it an incompatibility issue or something I am doing
    wrong? Below I am including my system's information, hopefully this
    will be useful for a diagnosis.

    The log file has the following:

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to localhost/127.0.0.1:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

    ================================================================================

    My system details:

    Ubuntu 10.04 LTS
    miguel@euler:~$ pig -version
    Apache Pig version 0.8.1 (r1094835)
    compiled Apr 18 2011, 19:26:53
    miguel@euler:~$ hadoop version
    Hadoop 0.20.203.0
    Subversion
    http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
    -r 1099333
    Compiled by oom on Wed May 4 07:57:50 PDT 2011

    My environment variables (I know I have more than I need but tutorials
    on the web have been a little hazy on this):

    miguel@euler:~$ printenv | grep "PIG\|HADOOP"
    PIG_DIR=/home/miguel/pig/pig-0.8.1
    PIG_HOME=/home/miguel/pig/pig-0.8.1
    HADOOP_HOME=/usr/local/hadoop
    PIGDIR=/home/miguel/pig/pig-0.8.1
    PIG_INSTALL=/home/miguel/pig/pig-0.8.1
    PIG_HADOOP_VERSION=20
    HADOOP_INSTALL=/usr/local/hadoop
    PIG_CLASSPATH=/usr/local/hadoop/conf
    HADOOPDIR=/usr/local/hadoop/conf

    Many thanks for any help,
    Rui Miguel Forte
  • Rui Miguel Forte at May 26, 2011 at 12:57 pm
    Hi Jonathan,

    Sorry or the late response, First of all thanks for your email. I'm
    afraid I still havent managed to get this to work. I am not clear on
    what you mean by register the antlr jar script though. How do I do
    this?

    Thanks,
    Miguel


    On 23 May 2011 20:35, Jonathan Coveney wrote:
    Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):

    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:

    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;

    2011/5/23 Rui Miguel Forte <ruimiguelforte@gmail.com>
    Hello,

    I have been trying to set up pig 0.8.1 to work with hadoop 0.20.203
    without success. At the moment, if I run pig -x local I correctly get
    access to the grunt shell but when I try to run the default mapreduce
    option it fails with error 2999, see below:

    miguel@euler:~$ pig
    2011-05-23 19:35:09,843 [main] INFO  org.apache.pig.Main - Logging
    error messages to: /home/miguel/pig_1306168509838.log
    2011-05-23 19:35:10,030 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting to hadoop file system at: hdfs://localhost:54310
    2011-05-23 19:35:10,168 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage
    Details at logfile: /home/miguel/pig_1306168509838.log

    Hadoop itself runs normally on my system. Any ideas on what to do to
    fix this? Is it an incompatibility issue or something I am doing
    wrong? Below I am including my system's information, hopefully this
    will be useful for a diagnosis.

    The log file has the following:

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to localhost/127.0.0.1:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

    ================================================================================

    My system details:

    Ubuntu 10.04 LTS
    miguel@euler:~$ pig -version
    Apache Pig version 0.8.1 (r1094835)
    compiled Apr 18 2011, 19:26:53
    miguel@euler:~$ hadoop version
    Hadoop 0.20.203.0
    Subversion
    http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
    -r 1099333
    Compiled by oom on Wed May  4 07:57:50 PDT 2011

    My environment variables (I know I have more than I need but tutorials
    on the web have been a little hazy on this):

    miguel@euler:~$ printenv | grep "PIG\|HADOOP"
    PIG_DIR=/home/miguel/pig/pig-0.8.1
    PIG_HOME=/home/miguel/pig/pig-0.8.1
    HADOOP_HOME=/usr/local/hadoop
    PIGDIR=/home/miguel/pig/pig-0.8.1
    PIG_INSTALL=/home/miguel/pig/pig-0.8.1
    PIG_HADOOP_VERSION=20
    HADOOP_INSTALL=/usr/local/hadoop
    PIG_CLASSPATH=/usr/local/hadoop/conf
    HADOOPDIR=/usr/local/hadoop/conf

    Many thanks for any help,
    Rui Miguel Forte
  • Jonathan Coveney at May 26, 2011 at 2:06 pm
    Did you build pig-withouthadoop.jar and did you make sure that it and your
    hadoop*.jar are on the classpath?

    If so, then when you make a pig script, begin it with: register
    /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;

    2011/5/26 Rui Miguel Forte <ruimiguelforte@gmail.com>
    Hi Jonathan,

    Sorry or the late response, First of all thanks for your email. I'm
    afraid I still havent managed to get this to work. I am not clear on
    what you mean by register the antlr jar script though. How do I do
    this?

    Thanks,
    Miguel


    On 23 May 2011 20:35, Jonathan Coveney wrote:
    Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):

    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I
    additionally
    manually registered the antlr jar:

    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;

    2011/5/23 Rui Miguel Forte <ruimiguelforte@gmail.com>
    Hello,

    I have been trying to set up pig 0.8.1 to work with hadoop 0.20.203
    without success. At the moment, if I run pig -x local I correctly get
    access to the grunt shell but when I try to run the default mapreduce
    option it fails with error 2999, see below:

    miguel@euler:~$ pig
    2011-05-23 19:35:09,843 [main] INFO org.apache.pig.Main - Logging
    error messages to: /home/miguel/pig_1306168509838.log
    2011-05-23 19:35:10,030 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting to hadoop file system at: hdfs://localhost:54310
    2011-05-23 19:35:10,168 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage
    Details at logfile: /home/miguel/pig_1306168509838.log

    Hadoop itself runs normally on my system. Any ideas on what to do to
    fix this? Is it an incompatibility issue or something I am doing
    wrong? Below I am including my system's information, hopefully this
    will be useful for a diagnosis.

    The log file has the following:

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to localhost/127.0.0.1:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at
    org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
    ================================================================================
    My system details:

    Ubuntu 10.04 LTS
    miguel@euler:~$ pig -version
    Apache Pig version 0.8.1 (r1094835)
    compiled Apr 18 2011, 19:26:53
    miguel@euler:~$ hadoop version
    Hadoop 0.20.203.0
    Subversion
    http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
    -r 1099333
    Compiled by oom on Wed May 4 07:57:50 PDT 2011

    My environment variables (I know I have more than I need but tutorials
    on the web have been a little hazy on this):

    miguel@euler:~$ printenv | grep "PIG\|HADOOP"
    PIG_DIR=/home/miguel/pig/pig-0.8.1
    PIG_HOME=/home/miguel/pig/pig-0.8.1
    HADOOP_HOME=/usr/local/hadoop
    PIGDIR=/home/miguel/pig/pig-0.8.1
    PIG_INSTALL=/home/miguel/pig/pig-0.8.1
    PIG_HADOOP_VERSION=20
    HADOOP_INSTALL=/usr/local/hadoop
    PIG_CLASSPATH=/usr/local/hadoop/conf
    HADOOPDIR=/usr/local/hadoop/conf

    Many thanks for any help,
    Rui Miguel Forte

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedMay 23, '11 at 4:50p
activeMay 26, '11 at 2:06p
posts4
users2
websitepig.apache.org

People

Translate

site design / logo © 2022 Grokbase