Grokbase Groups HBase user April 2011
FAQ
Hello guys

I am running cloudere distribution cdh3u0 on my cluster and i am trying
to connect pig with Hbase. I have 11 nodes on my cluster so i have
configured one machine as HBaseMaster and rest are Hbase Region Servers.
On my Master manshine:

1. Output of the command $*sudo jps is :*

6907 HMaster
14108 Jps
31470 QuorumPeerMain
22240 JobTracker
21602 SecondaryNameNode
21155 NameNode

2. Output of the command*$status 'detailed'*

-bash-3.2$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

hbase(main):001:0> *status 'detailed'*
version 0.90.1-cdh3u0
0 regionsInTransition
10 live servers
haisen2.ux.uis.no:60020 1303287066114
requests=0, regions=1, usedHeap=34, maxHeap=987
-ROOT-,,0
stores=1, storefiles=1, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen10.ux.uis.no:60020 1303287733515
requests=0, regions=1, usedHeap=35, maxHeap=987
task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
stores=1, storefiles=0, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen4.ux.uis.no:60020 1303287334273
requests=0, regions=0, usedHeap=33, maxHeap=987
haisen1.ux.uis.no:60020 1303643944066
requests=0, regions=0, usedHeap=35, maxHeap=987
haisen7.ux.uis.no:60020 1303287560071
requests=0, regions=1, usedHeap=32, maxHeap=987
test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
stores=1, storefiles=0, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen9.ux.uis.no:60020 1303287672340
requests=0, regions=1, usedHeap=24, maxHeap=987
.META.,,1
stores=1, storefiles=1, storefileSizeMB=0,
memstoreSizeMB=0,
storefi
leIndexSizeMB=0
haisen5.ux.uis.no:60020 1303287407405
requests=0, regions=0, usedHeap=28, maxHeap=987
haisen6.ux.uis.no:60020 1303287480842
requests=0, regions=0, usedHeap=23, maxHeap=987
haisen3.ux.uis.no:60020 1303287189985
requests=0, regions=0, usedHeap=27, maxHeap=987
haisen8.ux.uis.no:60020 1303287626373
requests=0, regions=0, usedHeap=30, maxHeap=987
0 dead servers


I can put data into HBase and pull it out just fine by hbase shell
command and i have a problem when i load data from hbase by using the
following Pig query:

*my_data = LOAD 'hbase://test' using org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as (a:chararray);dump my_data;*

i got a error message and my job was failure. If someone could help me to connect hbase with pig that would be great.


the problem I am having now is the following exception on the pig log file:

Backend error message
---------------------
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:156)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:145)
at
org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(HConnectionManager.java:989)
... 15 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 16 more

Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias my_data. Backend error : Unable to recreate
exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.pig.PigServer.openIterator(PigServer.java:742)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)




Thanks for helping


Byambajargal

Search Discussions

  • Sulabh choudhury at Apr 24, 2011 at 7:46 pm
    Have you tried loading like :-

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    *(a:bytearray)*;dump my_data;*
    or
    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a);dump my_data;*

    I am just trying to see if it works if you supply a different datatype ,
    bytearray instead of chararray, or without any datatype at all.
    On Sun, Apr 24, 2011 at 10:40 AM, byambajargal wrote:

    Hello guys

    I am running cloudere distribution cdh3u0 on my cluster and i am trying to
    connect pig with Hbase. I have 11 nodes on my cluster so i have configured
    one machine as HBaseMaster and rest are Hbase Region Servers.
    On my Master manshine:

    1. Output of the command $*sudo jps is :*

    6907 HMaster
    14108 Jps
    31470 QuorumPeerMain
    22240 JobTracker
    21602 SecondaryNameNode
    21155 NameNode

    2. Output of the command*$status 'detailed'*

    -bash-3.2$ hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

    hbase(main):001:0> *status 'detailed'*
    version 0.90.1-cdh3u0
    0 regionsInTransition
    10 live servers
    haisen2.ux.uis.no:60020 1303287066114
    requests=0, regions=1, usedHeap=34, maxHeap=987
    -ROOT-,,0
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen10.ux.uis.no:60020 1303287733515
    requests=0, regions=1, usedHeap=35, maxHeap=987
    task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen4.ux.uis.no:60020 1303287334273
    requests=0, regions=0, usedHeap=33, maxHeap=987
    haisen1.ux.uis.no:60020 1303643944066
    requests=0, regions=0, usedHeap=35, maxHeap=987
    haisen7.ux.uis.no:60020 1303287560071
    requests=0, regions=1, usedHeap=32, maxHeap=987
    test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen9.ux.uis.no:60020 1303287672340
    requests=0, regions=1, usedHeap=24, maxHeap=987
    .META.,,1
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen5.ux.uis.no:60020 1303287407405
    requests=0, regions=0, usedHeap=28, maxHeap=987
    haisen6.ux.uis.no:60020 1303287480842
    requests=0, regions=0, usedHeap=23, maxHeap=987
    haisen3.ux.uis.no:60020 1303287189985
    requests=0, regions=0, usedHeap=27, maxHeap=987
    haisen8.ux.uis.no:60020 1303287626373
    requests=0, regions=0, usedHeap=30, maxHeap=987
    0 dead servers


    I can put data into HBase and pull it out just fine by hbase shell command
    and i have a problem when i load data from hbase by using the following Pig
    query:

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a:chararray);dump my_data;*

    i got a error message and my job was failure. If someone could help me to
    connect hbase with pig that would be great.


    the problem I am having now is the following exception on the pig log file:

    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
    ... 15 more
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 16 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias my_data. Backend error : Unable to recreate
    exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.pig.PigServer.openIterator(PigServer.java:742)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)




    Thanks for helping


    Byambajargal

    --

    --
    Thanks and Regards,
    Sulabh Choudhury
  • Dmitriy Ryaboy at Apr 24, 2011 at 8:04 pm
    I suspect the problem here is that you don't have your hbase config
    directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
    environment variable.

    D
    On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury wrote:

    Have you tried loading like :-

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    *(a:bytearray)*;dump my_data;*
    or
    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a);dump my_data;*

    I am just trying to see if it works if you supply a different datatype ,
    bytearray instead of chararray, or without any datatype at all.

    On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <byambaa.0111@gmail.com
    wrote:
    Hello guys

    I am running cloudere distribution cdh3u0 on my cluster and i am trying to
    connect pig with Hbase. I have 11 nodes on my cluster so i have
    configured
    one machine as HBaseMaster and rest are Hbase Region Servers.
    On my Master manshine:

    1. Output of the command $*sudo jps is :*

    6907 HMaster
    14108 Jps
    31470 QuorumPeerMain
    22240 JobTracker
    21602 SecondaryNameNode
    21155 NameNode

    2. Output of the command*$status 'detailed'*

    -bash-3.2$ hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

    hbase(main):001:0> *status 'detailed'*
    version 0.90.1-cdh3u0
    0 regionsInTransition
    10 live servers
    haisen2.ux.uis.no:60020 1303287066114
    requests=0, regions=1, usedHeap=34, maxHeap=987
    -ROOT-,,0
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen10.ux.uis.no:60020 1303287733515
    requests=0, regions=1, usedHeap=35, maxHeap=987
    task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen4.ux.uis.no:60020 1303287334273
    requests=0, regions=0, usedHeap=33, maxHeap=987
    haisen1.ux.uis.no:60020 1303643944066
    requests=0, regions=0, usedHeap=35, maxHeap=987
    haisen7.ux.uis.no:60020 1303287560071
    requests=0, regions=1, usedHeap=32, maxHeap=987
    test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen9.ux.uis.no:60020 1303287672340
    requests=0, regions=1, usedHeap=24, maxHeap=987
    .META.,,1
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen5.ux.uis.no:60020 1303287407405
    requests=0, regions=0, usedHeap=28, maxHeap=987
    haisen6.ux.uis.no:60020 1303287480842
    requests=0, regions=0, usedHeap=23, maxHeap=987
    haisen3.ux.uis.no:60020 1303287189985
    requests=0, regions=0, usedHeap=27, maxHeap=987
    haisen8.ux.uis.no:60020 1303287626373
    requests=0, regions=0, usedHeap=30, maxHeap=987
    0 dead servers


    I can put data into HBase and pull it out just fine by hbase shell command
    and i have a problem when i load data from hbase by using the following Pig
    query:

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a:chararray);dump my_data;*

    i got a error message and my job was failure. If someone could help me to
    connect hbase with pig that would be great.


    the problem I am having now is the following exception on the pig log file:
    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
    ... 15 more
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 16 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias my_data. Backend error : Unable to recreate
    exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.pig.PigServer.openIterator(PigServer.java:742)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)



    Thanks for helping


    Byambajargal

    --

    --
    Thanks and Regards,
    Sulabh Choudhury
  • Bill Graham at Apr 24, 2011 at 9:46 pm
    I had this issue and had to add the HBase conf dir to HADOOP_CLASSPATH
    in conf/hadoop-env.sh on each of the nodes in the cluster so they
    could find Zookeeper.

    On Sun, Apr 24, 2011 at 1:04 PM, Dmitriy Ryaboy wrote:
    I suspect the problem here is that you don't have your hbase config
    directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
    environment variable.

    D
    On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhury wrote:

    Have you tried loading like :-

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    *(a:bytearray)*;dump my_data;*
    or
    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a);dump my_data;*

    I am just trying to see if it works if you supply a different datatype ,
    bytearray instead of chararray, or without any datatype at all.

    On Sun, Apr 24, 2011 at 10:40 AM, byambajargal <byambaa.0111@gmail.com
    wrote:
    Hello guys

    I am running cloudere distribution cdh3u0 on my cluster and i am trying to
    connect pig with Hbase. I have 11 nodes on my cluster so i have
    configured
    one machine as HBaseMaster and rest are Hbase Region Servers.
    On my Master manshine:

    1. Output of the command $*sudo jps is :*

    6907 HMaster
    14108 Jps
    31470 QuorumPeerMain
    22240 JobTracker
    21602 SecondaryNameNode
    21155 NameNode

    2. Output of the command*$status 'detailed'*

    -bash-3.2$ hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

    hbase(main):001:0> *status 'detailed'*
    version 0.90.1-cdh3u0
    0 regionsInTransition
    10 live servers
    haisen2.ux.uis.no:60020 1303287066114
    requests=0, regions=1, usedHeap=34, maxHeap=987
    -ROOT-,,0
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen10.ux.uis.no:60020 1303287733515
    requests=0, regions=1, usedHeap=35, maxHeap=987
    task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen4.ux.uis.no:60020 1303287334273
    requests=0, regions=0, usedHeap=33, maxHeap=987
    haisen1.ux.uis.no:60020 1303643944066
    requests=0, regions=0, usedHeap=35, maxHeap=987
    haisen7.ux.uis.no:60020 1303287560071
    requests=0, regions=1, usedHeap=32, maxHeap=987
    test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen9.ux.uis.no:60020 1303287672340
    requests=0, regions=1, usedHeap=24, maxHeap=987
    .META.,,1
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen5.ux.uis.no:60020 1303287407405
    requests=0, regions=0, usedHeap=28, maxHeap=987
    haisen6.ux.uis.no:60020 1303287480842
    requests=0, regions=0, usedHeap=23, maxHeap=987
    haisen3.ux.uis.no:60020 1303287189985
    requests=0, regions=0, usedHeap=27, maxHeap=987
    haisen8.ux.uis.no:60020 1303287626373
    requests=0, regions=0, usedHeap=30, maxHeap=987
    0 dead servers


    I can put data into HBase and pull it out just fine by hbase shell command
    and i have a problem when i load data from hbase by using the following Pig
    query:

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a:chararray);dump my_data;*

    i got a error message and my job was failure. If someone could help me to
    connect hbase with pig that would be great.


    the problem I am having now is the following exception on the pig log file:
    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
    ... 15 more
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 16 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias my_data. Backend error : Unable to recreate
    exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.pig.PigServer.openIterator(PigServer.java:742)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)



    Thanks for helping


    Byambajargal

    --

    --
    Thanks and Regards,
    Sulabh Choudhury
  • Byambajargal at Apr 25, 2011 at 10:51 am
    Thank you guys and Bill Graham

    I have solved the problem.
    I just add the following lines of shell to conf/hadoop-env.sh

    # if using HBase, likely want to include HBase config
    HBASE_CONF_DIR=${HBASE_CONF_DIR:-/etc/hbase/conf}
    if [ -n "$HBASE_CONF_DIR" ] && [ -d "$HBASE_CONF_DIR" ]; then
    export HADOOP_CLASSPATH=$HBASE_CONF_DIR:$HADOOP_CLASSPATH
    fi

    it works fine


    thanks

    Byambajargal


    On 4/24/11 23:45, Bill Graham wrote:
    I had this issue and had to add the HBase conf dir to HADOOP_CLASSPATH
    in conf/hadoop-env.sh on each of the nodes in the cluster so they
    could find Zookeeper.


    On Sun, Apr 24, 2011 at 1:04 PM, Dmitriy Ryaboywrote:
    I suspect the problem here is that you don't have your hbase config
    directory on Pig's classpath. You can add it by modifying the PIG_CLASSPATH
    environment variable.

    D

    On Sun, Apr 24, 2011 at 12:41 PM, sulabh choudhurywrote:
    Have you tried loading like :-

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    *(a:bytearray)*;dump my_data;*
    or
    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a);dump my_data;*

    I am just trying to see if it works if you supply a different datatype ,
    bytearray instead of chararray, or without any datatype at all.

    On Sun, Apr 24, 2011 at 10:40 AM, byambajargal<byambaa.0111@gmail.com
    wrote:
    Hello guys

    I am running cloudere distribution cdh3u0 on my cluster and i am trying to
    connect pig with Hbase. I have 11 nodes on my cluster so i have
    configured
    one machine as HBaseMaster and rest are Hbase Region Servers.
    On my Master manshine:

    1. Output of the command $*sudo jps is :*

    6907 HMaster
    14108 Jps
    31470 QuorumPeerMain
    22240 JobTracker
    21602 SecondaryNameNode
    21155 NameNode

    2. Output of the command*$status 'detailed'*

    -bash-3.2$ hbase shell
    HBase Shell; enter 'help<RETURN>' for list of supported commands.
    Type "exit<RETURN>" to leave the HBase Shell
    Version 0.90.1-cdh3u0, r, Fri Mar 25 16:10:51 PDT 2011

    hbase(main):001:0> *status 'detailed'*
    version 0.90.1-cdh3u0
    0 regionsInTransition
    10 live servers
    haisen2.ux.uis.no:60020 1303287066114
    requests=0, regions=1, usedHeap=34, maxHeap=987
    -ROOT-,,0
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen10.ux.uis.no:60020 1303287733515
    requests=0, regions=1, usedHeap=35, maxHeap=987
    task1,,1303326469811.418e9f137f5b1d33604419ef6a8acb6e.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen4.ux.uis.no:60020 1303287334273
    requests=0, regions=0, usedHeap=33, maxHeap=987
    haisen1.ux.uis.no:60020 1303643944066
    requests=0, regions=0, usedHeap=35, maxHeap=987
    haisen7.ux.uis.no:60020 1303287560071
    requests=0, regions=1, usedHeap=32, maxHeap=987
    test,,1303644907708.63ec7397e09d3335c8471aa90bcc5549.
    stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen9.ux.uis.no:60020 1303287672340
    requests=0, regions=1, usedHeap=24, maxHeap=987
    .META.,,1
    stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0,
    storefi
    leIndexSizeMB=0
    haisen5.ux.uis.no:60020 1303287407405
    requests=0, regions=0, usedHeap=28, maxHeap=987
    haisen6.ux.uis.no:60020 1303287480842
    requests=0, regions=0, usedHeap=23, maxHeap=987
    haisen3.ux.uis.no:60020 1303287189985
    requests=0, regions=0, usedHeap=27, maxHeap=987
    haisen8.ux.uis.no:60020 1303287626373
    requests=0, regions=0, usedHeap=30, maxHeap=987
    0 dead servers


    I can put data into HBase and pull it out just fine by hbase shell command
    and i have a problem when i load data from hbase by using the following Pig
    query:

    *my_data = LOAD 'hbase://test' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:a', '-caching 100') as
    (a:chararray);dump my_data;*

    i got a error message and my job was failure. If someone could help me to
    connect hbase with pig that would be great.


    the problem I am having now is the following exception on the pig log file:
    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:302)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989)
    ... 15 more
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 16 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias my_data. Backend error : Unable to recreate
    exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.hadoop.hbase.ZooKeeperConnectionException:
    org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.pig.PigServer.openIterator(PigServer.java:742)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)


    Thanks for helping


    Byambajargal
    --

    --
    Thanks and Regards,
    Sulabh Choudhury

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedApr 24, '11 at 5:40p
activeApr 25, '11 at 10:51a
posts5
users4
websitehbase.apache.org

People

Translate

site design / logo © 2018 Grokbase