FAQ
I sent this to pig apache user mailing list but have got no response.
Not sure if that list is still active.

thought I will post here if someone is able to help me.

I am in process of installing and learning pig. I have a hadoop
cluster and when I try to run pig in mapreduce mode it errors out:

Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:214)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
at org.apache.pig.PigServer.(PigServer.java:215)
at org.apache.pig.tools.grunt.Grunt.(Main.java:452)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

Search Discussions

  • Harsh J at May 26, 2011 at 5:01 pm
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.
    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Mohit Anchlia at May 26, 2011 at 5:05 pm
    For some reason I don't see that reply from Jonathan in my Inbox. I'll
    try to google it.

    What should be my next step in that case? I can't use pig then?
    On Thu, May 26, 2011 at 10:00 AM, Harsh J wrote:
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.
    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Jonathan Coveney at May 26, 2011 at 5:06 pm
    I'll repost it here then :)

    "Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):

    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:

    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"

    2011/5/26 Mohit Anchlia <mohitanchlia@gmail.com>
    For some reason I don't see that reply from Jonathan in my Inbox. I'll
    try to google it.

    What should be my next step in that case? I can't use pig then?
    On Thu, May 26, 2011 at 10:00 AM, Harsh J wrote:
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.
    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Mohit Anchlia at May 26, 2011 at 5:19 pm
    On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney wrote:
    I'll repost it here then :)

    "Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):
    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:
    Thanks Jonathan! I will give it a shot.
    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"
    Is this a windows command? Sorry, have not used this before.
    2011/5/26 Mohit Anchlia <mohitanchlia@gmail.com>
    For some reason I don't see that reply from Jonathan in my Inbox. I'll
    try to google it.

    What should be my next step in that case? I can't use pig then?
    On Thu, May 26, 2011 at 10:00 AM, Harsh J wrote:
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.

    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia <mohitanchlia@gmail.com>
    wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Mohit Anchlia at May 26, 2011 at 5:56 pm
    I've built pig-withouthadoop.jar and have copied it to my linux box.
    Now how do I put hadoop-core-0.20.203.0.jar and pig-withouthadoop.jar
    in the classpath. Is it by using CLASSPATH variable?
    On Thu, May 26, 2011 at 10:18 AM, Mohit Anchlia wrote:
    On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney wrote:
    I'll repost it here then :)

    "Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):
    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:
    Thanks Jonathan! I will give it a shot.
    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"
    Is this a windows command? Sorry, have not used this before.
    2011/5/26 Mohit Anchlia <mohitanchlia@gmail.com>
    For some reason I don't see that reply from Jonathan in my Inbox. I'll
    try to google it.

    What should be my next step in that case? I can't use pig then?
    On Thu, May 26, 2011 at 10:00 AM, Harsh J wrote:
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.

    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia <mohitanchlia@gmail.com>
    wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Mohit Anchlia at May 26, 2011 at 6:21 pm
    I added to PIG_CLASSPATH and went past the error but now I get a
    different error. Looks like I need to add some other jars but not sure
    which one.

    export PIG_CLASSPATH=$HADOOP_CONF_DIR:$HADOOP_HOME/hadoop-core-0.20.203.0.jar:$PIG_HOME/../pig-withouthadoop.jar

    ERROR 2998: Unhandled internal error.
    org/apache/commons/configuration/Configuration

    java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.(DefaultMetricsSystem.java:34)
    at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
    at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
    at org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:1319)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:109)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:196)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:116)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:187)
    at org.apache.pig.PigServer.(PigServer.java:228)
    at org.apache.pig.tools.grunt.Grunt.(Main.java:484)
    at org.apache.pig.Main.main(Main.java:108)
    Caused by: java.lang.ClassNotFoundException:
    org.apache.commons.configuration.Configuration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

    On Thu, May 26, 2011 at 10:55 AM, Mohit Anchlia wrote:
    I've built pig-withouthadoop.jar and have copied it to my linux box.
    Now how do I put hadoop-core-0.20.203.0.jar and pig-withouthadoop.jar
    in the classpath. Is it by using CLASSPATH variable?
    On Thu, May 26, 2011 at 10:18 AM, Mohit Anchlia wrote:
    On Thu, May 26, 2011 at 10:06 AM, Jonathan Coveney wrote:
    I'll repost it here then :)

    "Here is what I had to do to get pig running with a different version of
    Hadoop (in my case, the cloudera build but I'd try this as well):
    build pig-withouthadoop.jar by running "ant jar-withouthadoop". Then, when
    you run pig, put the pig-withouthadoop.jar on your classpath as well as your
    hadoop jar. In my case, I found that scripts only worked if I additionally
    manually registered the antlr jar:
    Thanks Jonathan! I will give it a shot.
    register /path/to/pig/build/ivy/lib/Pig/antlr-runtime-3.2.jar;"
    Is this a windows command? Sorry, have not used this before.
    2011/5/26 Mohit Anchlia <mohitanchlia@gmail.com>
    For some reason I don't see that reply from Jonathan in my Inbox. I'll
    try to google it.

    What should be my next step in that case? I can't use pig then?
    On Thu, May 26, 2011 at 10:00 AM, Harsh J wrote:
    I think Jonathan Coveney's reply on user@pig answered your question.
    Its basically an issue of hadoop version differences between the one
    Pig 0.8.1 release got bundled with vs. Hadoop 0.20.203 release which
    is newer.

    On Thu, May 26, 2011 at 10:26 PM, Mohit Anchlia <mohitanchlia@gmail.com>
    wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


    --
    Harsh J
  • Mohit Anchlia at May 26, 2011 at 7:01 pm
    I added all the jars in the classpath in HADOOP_HOME/lib and now I get
    to the grunt prompt. Will try the tutorials and see how it behaves :)

    Thanks for your help!
    On Thu, May 26, 2011 at 9:56 AM, Mohit Anchlia wrote:
    I sent this to pig apache user mailing list but have got no response.
    Not sure if that list is still active.

    thought I will post here if someone is able to help me.

    I am in process of installing and learning pig. I have a hadoop
    cluster and when I try to run pig in mapreduce mode it errors out:

    Hadoop version is hadoop-0.20.203.0 and pig version is pig-0.8.1

    Error before Pig is launched
    ----------------------------
    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to dsdb1/172.18.60.96:54310
    failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 26, '11 at 4:56p
activeMay 26, '11 at 7:01p
posts8
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase