FAQ
Hi

Iam using
Hadoop version : hadoop-0.20.2-cdh3u0 and PIG : pig-0.8.1


For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves) and
another VM 10.0.0.64 in which I ve installed PIG from which I run the PIG
commands.
I ve updated the xmls , please find the below
mapred site.xml
--------------
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>10.0.0.61:8021</value>
</property>
</configuration>

core-site.xml
----------
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://10.0.0.61:8020</value>
</property>
</configuration>

Hdfs-site.xml
----------------
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://10.0.0.61:8020</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>10.0.0.61:8021</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

masters
---------
10.0.0.61

slaves
--------
10.0.0.62
10.0.0.63

Have started start-all.sh. and checked with jps command in master(10.0.0.61)
I got response as

4313 Jps
3357 NameNode
3580 JobTracker
3502 SecondaryNameNode

and in slave I got

2662 DataNode
2902 Jps
2759 TaskTracker

So the Cluster is working...

But When I tried to connect to PIG i.e. bin/pig (I ve installed PIG in
10.0.0.64 this is out of Hadoop Cluster).
I am getting below exception again. Please help close this.

[kiranprasad.g@pig1 pig-0.8.1]$ bin/pig
2011-09-09 17:13:29,507 [main] INFO org.apache.pig.Main - Logging error
messages to: /home/kiranprasad.g/pig-0.8.1/pig_1315568609504.log
2011-09-09 17:13:29,676 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://10.0.0.61:8020
2011-09-09 17:13:29,804 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Failed to create DataStorage


The Error Log shows :

ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:214)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
at org.apache.pig.PigServer.(PigServer.java:215)
at org.apache.pig.tools.grunt.Grunt.(Main.java:452)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Call to /10.0.0.61:8020 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:170)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
================================================================================



Regards
Kiran.G

IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033
M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com

Search Discussions

  • Alan Gates at Sep 9, 2011 at 5:09 pm
    CDH 3 version of Hadoop uses a different version of HDFS then the client packaged with Pig 0.8.1. You should use the CDH version of Pig as well to make them work together.

    Alan.
    On Sep 9, 2011, at 2:44 AM, kiranprasad wrote:

    Hi

    Iam using
    Hadoop version : hadoop-0.20.2-cdh3u0 and PIG : pig-0.8.1


    For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves) and
    another VM 10.0.0.64 in which I ve installed PIG from which I run the PIG
    commands.
    I ve updated the xmls , please find the below
    mapred site.xml
    --------------
    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    </configuration>

    core-site.xml
    ----------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    </configuration>

    Hdfs-site.xml
    ----------------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>2</value>
    </property>

    masters
    ---------
    10.0.0.61

    slaves
    --------
    10.0.0.62
    10.0.0.63

    Have started start-all.sh. and checked with jps command in master(10.0.0.61)
    I got response as

    4313 Jps
    3357 NameNode
    3580 JobTracker
    3502 SecondaryNameNode

    and in slave I got

    2662 DataNode
    2902 Jps
    2759 TaskTracker

    So the Cluster is working...

    But When I tried to connect to PIG i.e. bin/pig (I ve installed PIG in
    10.0.0.64 this is out of Hadoop Cluster).
    I am getting below exception again. Please help close this.

    [kiranprasad.g@pig1 pig-0.8.1]$ bin/pig
    2011-09-09 17:13:29,507 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/kiranprasad.g/pig-0.8.1/pig_1315568609504.log
    2011-09-09 17:13:29,676 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
    to hadoop file system at: hdfs://10.0.0.61:8020
    2011-09-09 17:13:29,804 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage


    The Error Log shows :

    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to /10.0.0.61:8020 failed on local exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
    ================================================================================



    Regards
    Kiran.G

    IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033
    M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com
  • Keren Ouaknine at Sep 9, 2011 at 10:35 pm
    1. Can I generate pigperf on pig cdh3 too or will it only work on apache?
    2. pigmix2.patch according to Jira only works on pig 0.7 or will it work on
    pig apache 0.9 too?

    Thanks.
    On Fri, Sep 9, 2011 at 1:09 PM, Alan Gates wrote:

    CDH 3 version of Hadoop uses a different version of HDFS then the client
    packaged with Pig 0.8.1. You should use the CDH version of Pig as well to
    make them work together.

    Alan.
    On Sep 9, 2011, at 2:44 AM, kiranprasad wrote:

    Hi

    Iam using
    Hadoop version : hadoop-0.20.2-cdh3u0 and PIG : pig-0.8.1


    For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 - Slaves) and
    another VM 10.0.0.64 in which I ve installed PIG from which I run the PIG
    commands.
    I ve updated the xmls , please find the below
    mapred site.xml
    --------------
    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    </configuration>

    core-site.xml
    ----------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    </configuration>

    Hdfs-site.xml
    ----------------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>2</value>
    </property>

    masters
    ---------
    10.0.0.61

    slaves
    --------
    10.0.0.62
    10.0.0.63

    Have started start-all.sh. and checked with jps command in
    master(10.0.0.61)
    I got response as

    4313 Jps
    3357 NameNode
    3580 JobTracker
    3502 SecondaryNameNode

    and in slave I got

    2662 DataNode
    2902 Jps
    2759 TaskTracker

    So the Cluster is working...

    But When I tried to connect to PIG i.e. bin/pig (I ve installed PIG in
    10.0.0.64 this is out of Hadoop Cluster).
    I am getting below exception again. Please help close this.

    [kiranprasad.g@pig1 pig-0.8.1]$ bin/pig
    2011-09-09 17:13:29,507 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/kiranprasad.g/pig-0.8.1/pig_1315568609504.log
    2011-09-09 17:13:29,676 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: hdfs://10.0.0.61:8020
    2011-09-09 17:13:29,804 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage


    The Error Log shows :

    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to /10.0.0.61:8020 failed on local
    exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
    ================================================================================


    Regards
    Kiran.G

    IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033
    M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com

    --
    Keren Ouaknine
    Cell: +972 54 2565404
    Web: www.kereno.com
  • Daniel Dai at Sep 10, 2011 at 1:19 am

    On Fri, Sep 9, 2011 at 3:35 PM, Keren Ouaknine wrote:

    1. Can I generate pigperf on pig cdh3 too or will it only work on apache?
    Yes

    2. pigmix2.patch according to Jira only works on pig 0.7 or will it work on
    pig apache 0.9 too?
    Yes

    Thanks.
    On Fri, Sep 9, 2011 at 1:09 PM, Alan Gates wrote:

    CDH 3 version of Hadoop uses a different version of HDFS then the client
    packaged with Pig 0.8.1. You should use the CDH version of Pig as well to
    make them work together.

    Alan.
    On Sep 9, 2011, at 2:44 AM, kiranprasad wrote:

    Hi

    Iam using
    Hadoop version : hadoop-0.20.2-cdh3u0 and PIG : pig-0.8.1


    For Cluster I have 3 VMs(10.0.0.61-master, 10.0.0.62,10.0.0.63 -
    Slaves)
    and
    another VM 10.0.0.64 in which I ve installed PIG from which I run the
    PIG
    commands.
    I ve updated the xmls , please find the below
    mapred site.xml
    --------------
    <configuration>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    </configuration>

    core-site.xml
    ----------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    </configuration>

    Hdfs-site.xml
    ----------------
    <configuration>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.0.61:8020</value>
    </property>
    <property>
    <name>mapred.job.tracker</name>
    <value>10.0.0.61:8021</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>2</value>
    </property>

    masters
    ---------
    10.0.0.61

    slaves
    --------
    10.0.0.62
    10.0.0.63

    Have started start-all.sh. and checked with jps command in
    master(10.0.0.61)
    I got response as

    4313 Jps
    3357 NameNode
    3580 JobTracker
    3502 SecondaryNameNode

    and in slave I got

    2662 DataNode
    2902 Jps
    2759 TaskTracker

    So the Cluster is working...

    But When I tried to connect to PIG i.e. bin/pig (I ve installed PIG in
    10.0.0.64 this is out of Hadoop Cluster).
    I am getting below exception again. Please help close this.

    [kiranprasad.g@pig1 pig-0.8.1]$ bin/pig
    2011-09-09 17:13:29,507 [main] INFO org.apache.pig.Main - Logging
    error
    messages to: /home/kiranprasad.g/pig-0.8.1/pig_1315568609504.log
    2011-09-09 17:13:29,676 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: hdfs://10.0.0.61:8020
    2011-09-09 17:13:29,804 [main] ERROR org.apache.pig.Main - ERROR 2999:
    Unexpected internal error. Failed to create DataStorage


    The Error Log shows :

    ERROR 2999: Unexpected internal error. Failed to create DataStorage

    java.lang.RuntimeException: Failed to create DataStorage
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:214)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:134)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:183)
    at org.apache.pig.PigServer.<init>(PigServer.java:226)
    at org.apache.pig.PigServer.<init>(PigServer.java:215)
    at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55)
    at org.apache.pig.Main.run(Main.java:452)
    at org.apache.pig.Main.main(Main.java:107)
    Caused by: java.io.IOException: Call to /10.0.0.61:8020 failed on
    local
    exception: java.io.EOFException
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
    at org.apache.hadoop.ipc.Client.call(Client.java:743)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy0.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
    at
    org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
    at
    org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at
    org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
    at
    org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
    ... 9 more
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)
    ================================================================================


    Regards
    Kiran.G

    IMImobile Plot 770, Rd. 44 Jubilee Hills, Hyderabad - 500033
    M +91 9000170909 T +91 40 2355 5945 - Ext: 229 www.imimobile.com

    --
    Keren Ouaknine
    Cell: +972 54 2565404
    Web: www.kereno.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedSep 9, '11 at 9:44a
activeSep 10, '11 at 1:19a
posts4
users4
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase