Grokbase Groups Pig user July 2011
FAQ
Hi,
I have a latest pig build from trunk. I have configured it to run on 12-node
hadoop cluster.
I am trying to access a hbase table, the map job is running fine for
sometime...but after some time.. I am getting zookeeper exception

hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
2011-07-05 12:40:24,047 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://ub13:54310
2011-07-05 12:40:24,241 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to map-reduce job tracker at: ub13:54311

grunt> ht = load 'hbase://tweetinfo' using
org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
'-loadKey') as (id,name);
grunt> dump ht | HEAD 100;

2011-07-05 12:40:38,234 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
UNKNOWN
2011-07-05 12:40:38,481 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2011-07-05 12:40:38,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2011-07-05 12:40:38,514 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2011-07-05 12:40:38,559 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
the job
2011-07-05 12:40:38,572 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:host.name=ub13
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.version=1.6.0_20
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Sun Microsystems Inc.
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/tmp
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=i386
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:os.version=2.6.35-22-generic-pae
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.name=hadoop
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/home/hadoop
2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/usr/local/hadoop/hbase/conf
2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:38,791 [main-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
180000
2011-07-05 12:40:43,484 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2011-07-05 12:40:43,529 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2011-07-05 12:40:44,030 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
180000
2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=hconnection
2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
localhost/127.0.0.1:2181
2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
localhost/127.0.0.1:2181, initiating session
2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
180000
2011-07-05 12:40:45,964 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201107051228_0001
2011-07-05 12:40:45,964 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
2011-07-05 12:41:04,624 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1% complete
2011-07-05 12:41:07,638 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 3% complete
2011-07-05 12:41:08,141 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 4% complete
2011-07-05 12:41:10,652 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 7% complete
2011-07-05 12:41:11,155 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 9% complete
2011-07-05 12:41:16,681 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 12% complete
2011-07-05 12:41:17,183 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 13% complete
2011-07-05 12:41:19,933 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 15% complete
2011-07-05 12:41:22,444 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 18% complete
2011-07-05 12:41:22,947 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 19% complete
2011-07-05 12:41:26,463 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 21% complete
2011-07-05 12:41:28,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 24% complete
2011-07-05 12:41:29,478 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2011-07-05 12:41:31,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 27% complete
2011-07-05 12:41:37,516 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 28% complete
2011-07-05 12:41:43,544 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 30% complete
2011-07-05 12:41:56,124 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2011-07-05 12:41:58,635 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 33% complete
2011-07-05 12:41:59,140 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 34% complete
2011-07-05 12:42:02,155 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 36% complete
2011-07-05 12:42:10,700 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_201107051228_0001 has failed! Stop running all dependent jobs
2011-07-05 12:42:10,701 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-07-05 12:42:10,748 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
2011-07-05 12:42:10,748 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-07-05 12:42:10,750 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
2011-07-05 12:42:10 UNKNOWN

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_201107051228_0001 ht MAP_ONLY Message: Job failed!
hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,

Input(s):
Failed to read data from "hbase://tweetinfo"

Output(s):
Failed to produce result in
"hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201107051228_0001


2011-07-05 12:42:10,750 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log

LOG file details.....


Backend error message
---------------------
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(HConnectionManager.java:990)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.(HConnectionManager.java:156)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:146)
at
org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 12 more

Pig Stack Trace
---------------
ERROR 2997: Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias ht. Backend error : Unable to recreate exception
from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
HBase is able to connect to ZooKeeper but the connection closes immediately.
This could be a sign that the server has too many connections (30 is the
default). Consider inspecting your ZK server logs for that error and then
make sure you are reusing HBaseConfiguration as often as you can. See
HTable's javadoc for more information.
at org.apache.pig.PigServer.openIterator(PigServer.java:890)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
at org.apache.pig.Main.run(Main.java:487)
at org.apache.pig.Main.main(Main.java:108)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
Unable to recreate exception from backed error:
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
connect to ZooKeeper but the connection closes immediately. This could be a
sign that the server has too many connections (30 is the default). Consider
inspecting your ZK server logs for that error and then make sure you are
reusing HBaseConfiguration as often as you can. See HTable's javadoc for
more information.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
at org.apache.pig.PigServer.storeEx(PigServer.java:995)
at org.apache.pig.PigServer.store(PigServer.java:962)
at org.apache.pig.PigServer.openIterator(PigServer.java:875)
... 7 more
================================================================================


My hbase-site.xml looks like this

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ub13:54310/hbase</value>
<final>true</final>
</property>


<property>
<name>hbase.zookeeper.quorum</name>
<value>ub13,ub12,ub11</value>
<final>true</final>
</property>

<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>200</value>
<final>true</final>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum
</description>
</property>


Zookeeper log -- http://pastebin.com/RRdzZYui

I have increased the maximum number of zookeeper connections.. but still I
am getting the same error.. I am not able to get enough out of zookeeper
logs also.. its just saying.. end of stream exception..socket ended..
something like that..
The table I am trying to access consists of around 21 lacs record..
If I try to access small tables.. I am able to access it..

What can be the issue ?

Thanks,
Praveenesh

Search Discussions

  • Dmitriy Ryaboy at Jul 5, 2011 at 9:16 pm
    What version of HBase are you running?

    On Tue, Jul 5, 2011 at 1:45 AM, praveenesh kumar wrote:
    Hi,
    I have a latest pig build from trunk. I have configured it to run on 12-node
    hadoop cluster.
    I am trying to access a hbase table, the map job is running fine for
    sometime...but after some time.. I am getting zookeeper exception

    hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
    log4j:WARN No appenders could be found for logger
    (org.apache.hadoop.conf.Configuration).
    log4j:WARN Please initialize the log4j system properly.
    2011-07-05 12:40:23,737 [main] INFO  org.apache.pig.Main - Logging error
    messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
    2011-07-05 12:40:24,047 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
    to hadoop file system at: hdfs://ub13:54310
    2011-07-05 12:40:24,241 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
    to map-reduce job tracker at: ub13:54311

    grunt> ht = load 'hbase://tweetinfo' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
    '-loadKey') as (id,name);
    grunt> dump ht | HEAD 100;

    2011-07-05 12:40:38,234 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
    UNKNOWN
    2011-07-05 12:40:38,481 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
    File concatenation threshold: 100 optimistic? false
    2011-07-05 12:40:38,513 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size before optimization: 1
    2011-07-05 12:40:38,514 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size after optimization: 1
    2011-07-05 12:40:38,559 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
    the job
    2011-07-05 12:40:38,572 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:host.name=ub13
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.version=1.6.0_20
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.vendor=Sun Microsystems Inc.
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.io.tmpdir=/tmp
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:java.compiler=<NA>
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:os.name=Linux
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:os.arch=i386
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:os.version=2.6.35-22-generic-pae
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:user.name=hadoop
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:user.home=/home/hadoop
    2011-07-05 12:40:38,737 [main] INFO  org.apache.zookeeper.ZooKeeper - Client
    environment:user.dir=/usr/local/hadoop/hbase/conf
    2011-07-05 12:40:38,739 [main] INFO  org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:38,791 [main-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
    180000
    2011-07-05 12:40:43,484 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting up single store job
    2011-07-05 12:40:43,529 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1 map-reduce job(s) waiting for submission.
    2011-07-05 12:40:44,030 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 0% complete
    2011-07-05 12:40:44,499 [Thread-13] INFO  org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
    180000
    2011-07-05 12:40:44,554 [Thread-13] INFO  org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
    180000
    2011-07-05 12:40:45,964 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - HadoopJobId: job_201107051228_0001
    2011-07-05 12:40:45,964 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - More information at:
    http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
    2011-07-05 12:41:04,624 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1% complete
    2011-07-05 12:41:07,638 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 3% complete
    2011-07-05 12:41:08,141 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 4% complete
    2011-07-05 12:41:10,652 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 7% complete
    2011-07-05 12:41:11,155 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 9% complete
    2011-07-05 12:41:16,681 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 12% complete
    2011-07-05 12:41:17,183 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 13% complete
    2011-07-05 12:41:19,933 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 15% complete
    2011-07-05 12:41:22,444 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 18% complete
    2011-07-05 12:41:22,947 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 19% complete
    2011-07-05 12:41:26,463 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 21% complete
    2011-07-05 12:41:28,473 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 24% complete
    2011-07-05 12:41:29,478 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 25% complete
    2011-07-05 12:41:31,488 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 27% complete
    2011-07-05 12:41:37,516 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 28% complete
    2011-07-05 12:41:43,544 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 30% complete
    2011-07-05 12:41:56,124 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 31% complete
    2011-07-05 12:41:58,635 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 33% complete
    2011-07-05 12:41:59,140 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 34% complete
    2011-07-05 12:42:02,155 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 36% complete
    2011-07-05 12:42:10,700 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - job job_201107051228_0001 has failed! Stop running all dependent jobs
    2011-07-05 12:42:10,701 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 100% complete
    2011-07-05 12:42:10,748 [main] ERROR
    org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
    recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    2011-07-05 12:42:10,748 [main] ERROR
    org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
    2011-07-05 12:42:10,750 [main] INFO
    org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

    HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt
    Features
    0.20-append-for-hbase   0.10.0-SNAPSHOT hadoop  2011-07-05 12:40:38
    2011-07-05 12:42:10     UNKNOWN

    Failed!

    Failed Jobs:
    JobId   Alias   Feature Message Outputs
    job_201107051228_0001   ht      MAP_ONLY        Message: Job failed!
    hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,

    Input(s):
    Failed to read data from "hbase://tweetinfo"

    Output(s):
    Failed to produce result in
    "hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    job_201107051228_0001


    2011-07-05 12:42:10,750 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - Failed!
    2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log

    LOG file details.....


    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 12 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias ht. Backend error : Unable to recreate exception
    from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    HBase is able to connect to ZooKeeper but the connection closes immediately.
    This could be a sign that the server has too many connections (30 is the
    default). Consider inspecting your ZK server logs for that error and then
    make sure you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc for more information.
    at org.apache.pig.PigServer.openIterator(PigServer.java:890)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
    at org.apache.pig.Main.run(Main.java:487)
    at org.apache.pig.Main.main(Main.java:108)
    Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
    Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
    at
    org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
    at org.apache.pig.PigServer.storeEx(PigServer.java:995)
    at org.apache.pig.PigServer.store(PigServer.java:962)
    at org.apache.pig.PigServer.openIterator(PigServer.java:875)
    ... 7 more
    ================================================================================


    My hbase-site.xml looks like this

    <configuration>
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ub13:54310/hbase</value>
    <final>true</final>
    </property>


    <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ub13,ub12,ub11</value>
    <final>true</final>
    </property>

    <property>
    <name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>200</value>
    <final>true</final>
    </property>

    <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
    false: standalone and pseudo-distributed setups with managed Zookeeper
    true: fully-distributed with unmanaged Zookeeper Quorum
    </description>
    </property>


    Zookeeper log -- http://pastebin.com/RRdzZYui

    I have increased the maximum number of zookeeper connections.. but still I
    am getting the same error.. I am not able to get enough out of zookeeper
    logs also.. its just saying.. end of stream exception..socket ended..
    something like that..
    The table I am trying to access consists of around 21 lacs record..
    If I try to access small tables.. I am able to access it..

    What can be the issue ?

    Thanks,
    Praveenesh
  • Praveenesh kumar at Jul 6, 2011 at 4:37 am
    I am using HBase 0.90.3
    On Wed, Jul 6, 2011 at 2:45 AM, Dmitriy Ryaboy wrote:

    What version of HBase are you running?

    On Tue, Jul 5, 2011 at 1:45 AM, praveenesh kumar wrote:
    Hi,
    I have a latest pig build from trunk. I have configured it to run on 12-node
    hadoop cluster.
    I am trying to access a hbase table, the map job is running fine for
    sometime...but after some time.. I am getting zookeeper exception

    hadoop@ub13:/usr/local/hadoop/hbase/conf$ pig
    log4j:WARN No appenders could be found for logger
    (org.apache.hadoop.conf.Configuration).
    log4j:WARN Please initialize the log4j system properly.
    2011-07-05 12:40:23,737 [main] INFO org.apache.pig.Main - Logging error
    messages to: /usr/local/hadoop/hbase/conf/pig_1309849823721.log
    2011-07-05 12:40:24,047 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to hadoop file system at: hdfs://ub13:54310
    2011-07-05 12:40:24,241 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting
    to map-reduce job tracker at: ub13:54311

    grunt> ht = load 'hbase://tweetinfo' using
    org.apache.pig.backend.hadoop.hbase.HBaseStorage('prim_tweetinfo:name',
    '-loadKey') as (id,name);
    grunt> dump ht | HEAD 100;

    2011-07-05 12:40:38,234 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script:
    UNKNOWN
    2011-07-05 12:40:38,481 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
    File concatenation threshold: 100 optimistic? false
    2011-07-05 12:40:38,513 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size before optimization: 1
    2011-07-05 12:40:38,514 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size after optimization: 1
    2011-07-05 12:40:38,559 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to
    the job
    2011-07-05 12:40:38,572 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:host.name=ub13
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:java.version=1.6.0_20
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:java.vendor=Sun Microsystems Inc.
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client

    environment:java.class.path=/usr/local/hadoop/pig-0.10/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/usr/local/hadoop/pig-0.10/bin/../build/classes:/usr/local/hadoop/pig-0.10/bin/../build/test/classes:/usr/local/hadoop/pig-0.10/bin/../pig-*-core.jar:/usr/local/hadoop/pig-0.10/bin/../build/pig-0.10.0-SNAPSHOT.jar:/usr/local/hadoop/pig-0.10/bin/../lib/automaton.jar:/usr/local/hadoop/hadoop/hadoop-0.20-append-for-hbase-core.jar:/usr/local/hadoop/hbase/hbase-0.90.3.jar:/usr/local/hadoop/hbase/hbase-0.90.3-tests.jar:/usr/local/hadoop/hbase/lib/zookeeper-3.3.2.jar:/usr/local/hadoop/hadoop:/usr/local/hadoop/hadoop/conf
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client

    environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:java.io.tmpdir=/tmp
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:java.compiler=<NA>
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:os.name=Linux
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:os.arch=i386
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:os.version=2.6.35-22-generic-pae
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:user.name=hadoop
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:user.home=/home/hadoop
    2011-07-05 12:40:38,737 [main] INFO org.apache.zookeeper.ZooKeeper - Client
    environment:user.dir=/usr/local/hadoop/hbase/conf
    2011-07-05 12:40:38,739 [main] INFO org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:38,791 [main-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:38,814 [main-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:38,833 [main-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c0009, negotiated timeout =
    180000
    2011-07-05 12:40:43,484 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting up single store job
    2011-07-05 12:40:43,529 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1 map-reduce job(s) waiting for submission.
    2011-07-05 12:40:44,030 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 0% complete
    2011-07-05 12:40:44,499 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:44,500 [Thread-13-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:44,501 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:44,512 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000a, negotiated timeout =
    180000
    2011-07-05 12:40:44,554 [Thread-13] INFO org.apache.zookeeper.ZooKeeper -
    Initiating client connection, connectString=localhost:2181
    sessionTimeout=180000 watcher=hconnection
    2011-07-05 12:40:44,556 [Thread-13-SendThread()] INFO
    org.apache.zookeeper.ClientCnxn - Opening socket connection to server
    localhost/127.0.0.1:2181
    2011-07-05 12:40:44,558 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Socket connection established to
    localhost/127.0.0.1:2181, initiating session
    2011-07-05 12:40:44,569 [Thread-13-SendThread(localhost:2181)] INFO
    org.apache.zookeeper.ClientCnxn - Session establishment complete on server
    localhost/127.0.0.1:2181, sessionid = 0x30f91a0a2c000b, negotiated timeout =
    180000
    2011-07-05 12:40:45,964 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - HadoopJobId: job_201107051228_0001
    2011-07-05 12:40:45,964 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - More information at:
    http://ub13:50030/jobdetails.jsp?jobid=job_201107051228_0001
    2011-07-05 12:41:04,624 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1% complete
    2011-07-05 12:41:07,638 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 3% complete
    2011-07-05 12:41:08,141 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 4% complete
    2011-07-05 12:41:10,652 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 7% complete
    2011-07-05 12:41:11,155 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 9% complete
    2011-07-05 12:41:16,681 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 12% complete
    2011-07-05 12:41:17,183 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 13% complete
    2011-07-05 12:41:19,933 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 15% complete
    2011-07-05 12:41:22,444 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 18% complete
    2011-07-05 12:41:22,947 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 19% complete
    2011-07-05 12:41:26,463 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 21% complete
    2011-07-05 12:41:28,473 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 24% complete
    2011-07-05 12:41:29,478 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 25% complete
    2011-07-05 12:41:31,488 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 27% complete
    2011-07-05 12:41:37,516 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 28% complete
    2011-07-05 12:41:43,544 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 30% complete
    2011-07-05 12:41:56,124 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 31% complete
    2011-07-05 12:41:58,635 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 33% complete
    2011-07-05 12:41:59,140 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 34% complete
    2011-07-05 12:42:02,155 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 36% complete
    2011-07-05 12:42:10,700 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - job job_201107051228_0001 has failed! Stop running all dependent jobs
    2011-07-05 12:42:10,701 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 100% complete
    2011-07-05 12:42:10,748 [main] ERROR
    org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
    recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    2011-07-05 12:42:10,748 [main] ERROR
    org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
    2011-07-05 12:42:10,750 [main] INFO
    org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    0.20-append-for-hbase 0.10.0-SNAPSHOT hadoop 2011-07-05 12:40:38
    2011-07-05 12:42:10 UNKNOWN

    Failed!

    Failed Jobs:
    JobId Alias Feature Message Outputs
    job_201107051228_0001 ht MAP_ONLY Message: Job failed!
    hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030,

    Input(s):
    Failed to read data from "hbase://tweetinfo"

    Output(s):
    Failed to produce result in
    "hdfs://ub13:54310/tmp/temp-1630311766/tmp888118030"

    Counters:
    Total records written : 0
    Total bytes written : 0
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    job_201107051228_0001


    2011-07-05 12:42:10,750 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - Failed!
    2011-07-05 12:42:10,783 [main] ERROR org.apache.pig.tools.grunt.Grunt -
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    Details at logfile: /usr/local/hadoop/hbase/conf/pig_1309849823721.log

    LOG file details.....


    Backend error message
    ---------------------
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:990)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:303)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:294)
    at
    org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:168)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:146)
    at
    org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:457)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
    KeeperErrorCode = ConnectionLoss for /hbase
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at
    org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
    at
    org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
    at
    org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
    ... 12 more

    Pig Stack Trace
    ---------------
    ERROR 2997: Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.

    org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
    open iterator for alias ht. Backend error : Unable to recreate exception
    from backed error: org.apache.hadoop.hbase.ZooKeeperConnectionException:
    HBase is able to connect to ZooKeeper but the connection closes
    immediately.
    This could be a sign that the server has too many connections (30 is the
    default). Consider inspecting your ZK server logs for that error and then
    make sure you are reusing HBaseConfiguration as often as you can. See
    HTable's javadoc for more information.
    at org.apache.pig.PigServer.openIterator(PigServer.java:890)
    at
    org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:655)
    at
    org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
    at
    org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
    at org.apache.pig.Main.run(Main.java:487)
    at org.apache.pig.Main.main(Main.java:108)
    Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997:
    Unable to recreate exception from backed error:
    org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
    connect to ZooKeeper but the connection closes immediately. This could be a
    sign that the server has too many connections (30 is the default). Consider
    inspecting your ZK server logs for that error and then make sure you are
    reusing HBaseConfiguration as often as you can. See HTable's javadoc for
    more information.
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
    at
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:341)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1313)
    at
    org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1298)
    at org.apache.pig.PigServer.storeEx(PigServer.java:995)
    at org.apache.pig.PigServer.store(PigServer.java:962)
    at org.apache.pig.PigServer.openIterator(PigServer.java:875)
    ... 7 more
    ================================================================================

    My hbase-site.xml looks like this

    <configuration>
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://ub13:54310/hbase</value>
    <final>true</final>
    </property>


    <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ub13,ub12,ub11</value>
    <final>true</final>
    </property>

    <property>
    <name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>200</value>
    <final>true</final>
    </property>

    <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>The mode the cluster will be in. Possible values are
    false: standalone and pseudo-distributed setups with managed Zookeeper
    true: fully-distributed with unmanaged Zookeeper Quorum
    </description>
    </property>


    Zookeeper log -- http://pastebin.com/RRdzZYui

    I have increased the maximum number of zookeeper connections.. but still I
    am getting the same error.. I am not able to get enough out of zookeeper
    logs also.. its just saying.. end of stream exception..socket ended..
    something like that..
    The table I am trying to access consists of around 21 lacs record..
    If I try to access small tables.. I am able to access it..

    What can be the issue ?

    Thanks,
    Praveenesh

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedJul 5, '11 at 8:45a
activeJul 6, '11 at 4:37a
posts3
users2
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase