[HBase-user] Region servers down when inserting with hbase0.20.0 rc

On Tue, Aug 4, 2009 at 1:23 AM, Zheng Lv wrote:

Hello Everyone,
We are testing hbase0.20.0 rc in a cluster with 5 nodes, inserting web
pages at a speed of 20 pages per second. A few minutes later one region
server shut down, and about 40 minutes later another shut down too, leaving
2 of 4 region servers running.
I note that the contents of logs in two servers are not similar, and
below is the contents.

ubuntu7:


2009-08-04 08:59:35,870 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249344005673,
entries=34,
calcsize=10262, filesize=7353. New hlog
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858
2009-08-04 09:12:49,462 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:12:49,564 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:12:49,959 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:50860 remote=ubuntu7/192.168.33.6:2222]
2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:12:49,965 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-08-04 09:13:39,460 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:13:39,567 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:13:39,807 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:13:46,192 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu9/192.168.33.5:2222
2009-08-04 09:13:52,184 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:13:52,185 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:13:52,923 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:49662 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:13:55,342 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Session Expired
at

org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:13:55,342 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-08-04 09:13:55,343 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-08-04 09:13:55,343 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region
Server
2009-08-04 09:13:55,375 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=1, stores=2, storefiles=2, storefileIndexSize=0,
memstoreSize=0, usedHeap=24, maxHeap=2993, blockCacheSize=5151568,
blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
2009-08-04 09:13:55,833 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.6:60020.logFlusher exiting
2009-08-04 09:13:55,852 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.6:60020.cacheFlusher exiting
2009-08-04 09:13:55,882 INFO
org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-08-04 09:13:58,340 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC
Server Responder
2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC
Server listener on 60020
2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-08-04 09:13:58,471 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.6:60020.compactor exiting
2009-08-04 09:13:58,472 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.6:60020.majorCompactionChecker exiting
2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closing leases
2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closed leases
2009-08-04 09:14:01,895 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-08-04 09:14:25,347 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:25,348 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 26 seconds
2009-08-04 09:14:25,348 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries
left
4
2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 27 seconds
2009-08-04 09:14:25,753 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries
left
3
2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 28 seconds
2009-08-04 09:14:26,557 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries
left
2
2009-08-04 09:14:28,161 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:28,162 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 29 seconds
2009-08-04 09:14:28,162 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries
left
1
2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at

org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at

org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2009-08-04 09:14:31,368 WARN org.apache.hadoop.hdfs.DFSClient: Could not
get
block locations. Source file
"/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858" -
Aborting...
2009-08-04 09:14:31,371 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at

org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at

sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at

org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at

org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at

org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:14:31,373 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed .META.,,1
2009-08-04 09:14:31,373 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.6:60020
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Session:
0x22e2b69b350000 closed
2009-08-04 09:14:31,374 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.6:60020 exiting
2009-08-04 09:14:31,377 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-08-04 09:14:31,391 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-08-04 09:14:31,392 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
complete
2009-08-04 09:14:31,394 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-08-04 09:14:31,394 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-08-04 09:14:31,397 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222
sessionTimeout=30000
wat[email protected]
2009-08-04 09:14:31,398 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:49672 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:14:31,417 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-08-04 09:14:31,447 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-08-04 09:14:31,447 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:14:31,448 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:14:31,452 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that
region server is shutting down at: 192.168.33.6:60020
2009-08-04 09:14:31,453 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed to send exiting
message to master:
java.lang.NullPointerException
at

org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:660)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:14:31,453 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at:
192.168.33.6:60020
2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x122e2b7548e0087
2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x122e2b7548e0087
2009-08-04 09:14:31,460 INFO org.apache.zookeeper.ClientCnxn: Exception
while closing send thread for session 0x122e2b7548e0087 : Read error rc =
-1
java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]








ubuntu9:

2009-08-04 08:59:40,655 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,,1249347647788
2009-08-04 08:59:40,656 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,,1249347647788
2009-08-04 08:59:40,683 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249347647788/491403556 available; sequence id is 0
2009-08-04 09:01:40,266 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090,
entries=3147, calcsize=29823872, filesize=29491785. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254
2009-08-04 09:01:40,269 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090 whose
highest sequence/edit id is 5984
2009-08-04 09:10:31,649 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254,
entries=6720, calcsize=63790080, filesize=63079139. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638
2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638,
entries=6715, calcsize=63794066, filesize=63090684. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324
2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254 whose
highest sequence/edit id is 12705
2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324,
entries=6710, calcsize=63797225, filesize=63093218. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979
2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638 whose
highest sequence/edit id is 19419
2009-08-04 09:12:40,827 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249347647788
2009-08-04 09:12:49,963 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249347647788 in 9sec
2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979,
entries=6720, calcsize=63800107, filesize=63095030. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060
2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324 whose
highest sequence/edit id is 26134
2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060,
entries=6715, calcsize=63798666, filesize=63094154. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426
2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979 whose
highest sequence/edit id is 32850
2009-08-04 09:14:56,116 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:14:57,075 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:44326 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:15:15,359 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249347647788
2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249347647788 in 5sec
2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,,1249347647788
2009-08-04 09:15:21,963 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249347647788
2009-08-04 09:15:24,791 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249348520980/935635885 available; sequence id is 45742
2009-08-04 09:15:24,792 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249348520980
2009-08-04 09:15:24,995 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380
available; sequence id is 45743
2009-08-04 09:15:24,996 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:25,039 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,,1249347647788', STARTKEY => '', ENDKEY => '', ENCODED =>
491403556, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}, {NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS =>
'1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}]}}, new regions: webpage,,1249348520980,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980.
Split took 4sec
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,,1249348520980
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,,1249348520980
2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249348520980/935635885 available; sequence id is 45742
2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249348520980
2009-08-04 09:15:26,690 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:26,998 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380
available; sequence id is 45743
2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249348520980 in 4sec
2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:16:00,015 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 0 on 60020 took 25134ms appending an edit to hlog;
editcount=6668
2009-08-04 09:16:00,016 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 19781ms optional sync'ing
hlog; editcount=6674
2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426,
entries=6710, calcsize=63779032, filesize=63075244. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377
2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060 whose
highest sequence/edit id is 39565
2009-08-04 09:16:30,001 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 1 on 60020 took 27826ms appending an edit to hlog;
editcount=204
2009-08-04 09:16:30,002 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 19986ms optional sync'ing
hlog; editcount=205
2009-08-04 09:16:38,451 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
at

org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255)
at

org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
2009-08-04 09:16:38,457 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call openScanner([[email protected], startRow=, stopRow=,
maxVersions=1, timeRange=[0,9223372036854775807),
families={(family=CF_CONTENT, columns={}), (family=CF_INFORMATION,
columns={}}) from 192.168.33.7:52275: error:
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
at

org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255)
at

org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377,
entries=6955, calcsize=66566006, filesize=65836364. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837
2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426 whose
highest sequence/edit id is 46277
2009-08-04 09:17:28,853 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377 whose
highest sequence/edit id is 53231
2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
in 1mins, 59sec
2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:17:30,218 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:17:33,371 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797
available; sequence id is 53366
2009-08-04 09:17:33,372 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:33,575 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105
available; sequence id is 53367
2009-08-04 09:17:33,576 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:34,580 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 0 time(s).
2009-08-04 09:17:35,581 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 1 time(s).
2009-08-04 09:17:36,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 2 time(s).
2009-08-04 09:17:37,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 3 time(s).
2009-08-04 09:17:38,583 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 4 time(s).
2009-08-04 09:17:39,584 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 5 time(s).
2009-08-04 09:17:40,585 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 6 time(s).
2009-08-04 09:17:41,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 7 time(s).
2009-08-04 09:17:42,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 8 time(s).
2009-08-04 09:17:43,587 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 9 time(s).
2009-08-04 09:17:45,611 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744',
ENDKEY => '', ENCODED => 1870783380, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989.
Split took 15sec
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,943 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797
available; sequence id is 53366
2009-08-04 09:17:47,944 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,944 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:48,163 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105
available; sequence id is 53367
2009-08-04 09:17:52,777 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
in 4sec
2009-08-04 09:17:52,778 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:18:01,313 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
in 8sec
2009-08-04 09:24:26,812 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837,
entries=6661, calcsize=63769148, filesize=63059485. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799
2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799,
entries=6661, calcsize=63769500, filesize=63059617. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458
2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837 whose
highest sequence/edit id is 59894
2009-08-04 09:25:34,681 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
in 4sec
2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:39,244 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000/1497560335
available; sequence id is 68239
2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000
2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000/1242271099
available; sequence id is 68240
2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000
2009-08-04 09:25:42,663 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168',
ENDKEY => '', ENCODED => 203332105, OFFLINE => true, SPLIT => true, TABLE
=>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000.
Split took 3sec
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:25,058 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035/2028268999
available; sequence id is 66248
2009-08-04 09:30:25,059 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:25,061 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:25,299 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035/1041724375
available; sequence id is 66249
2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
in 3sec
2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:41,191 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
in 12sec
2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458,
entries=6662, calcsize=63774480, filesize=63064760. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432
2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799 whose
highest sequence/edit id is 66556
2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432,
entries=6661, calcsize=63776160, filesize=63066278. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174
2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458 whose
highest sequence/edit id is 73217
2009-08-04 09:32:35,692 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
in 2sec
2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:38,479 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:40,510 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326/481931133
available; sequence id is 82900
2009-08-04 09:32:42,063 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326
2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637
available; sequence id is 82901
2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:42,298 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898',
ENDKEY => '', ENCODED => 1041724375, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS =>
'2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326.
Split took 3sec
2009-08-04 09:32:43,169 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:43,170 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637
available; sequence id is 82901
2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
in 11sec
2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:55,912 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:56,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115
available; sequence id is 83905
2009-08-04 09:32:56,631 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:56,793 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028
available; sequence id is 83906
2009-08-04 09:32:56,794 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:56,808 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298',
ENDKEY => '', ENCODED => 922277637, OFFLINE => true, SPLIT => true, TABLE
=>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293.
Split took 1sec
2009-08-04 09:32:58,205 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,206 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:58,206 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,304 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028
available; sequence id is 83906
2009-08-04 09:32:58,304 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:58,306 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,441 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115
available; sequence id is 83905
2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
in 5sec
2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:33:09,988 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
in 6sec
2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174,
entries=6658, calcsize=63774356, filesize=63065350. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082
2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432 whose
highest sequence/edit id is 79878
2009-08-04 09:34:22,724 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082,
entries=6666, calcsize=63777964, filesize=63067534. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713
2009-08-04 09:34:22,725 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174 whose
highest sequence/edit id is 86545
2009-08-04 09:35:06,800 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
in 2sec
2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:09,561 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:12,780 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334/316888594
available; sequence id is 98604
2009-08-04 09:35:12,781 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334
2009-08-04 09:35:12,949 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334/1414734782
available; sequence id is 98605
2009-08-04 09:35:12,950 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334
2009-08-04 09:35:12,966 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720',
ENDKEY => '', ENCODED => 112267028, OFFLINE => true, SPLIT => true, TABLE
=>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334.
Split took 3sec
2009-08-04 09:37:58,723 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:37:58,724 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:38:18,205 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622/1005978655
available; sequence id is 113675
2009-08-04 09:38:18,206 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:38:22,211 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
in 4sec
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732/671938285
available; sequence id is 174598
2009-08-04 09:47:32,694 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,913 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732/469485305
available; sequence id is 174599
2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
in 2sec
2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:46,699 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
in 11sec
2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713,
entries=6662, calcsize=63776160, filesize=63066462. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966
2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082 whose
highest sequence/edit id is 93207
2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966,
entries=6661, calcsize=63776160, filesize=63066278. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537
2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713 whose
highest sequence/edit id is 175865
2009-08-04 09:49:42,602 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:44,186 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537,
entries=6661, calcsize=63776160, filesize=63066258. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168
2009-08-04 09:49:44,187 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966 whose
highest sequence/edit id is 182525
2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
in 6sec
2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:49,199 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:49,954 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991
available; sequence id is 189657
2009-08-04 09:49:49,955 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945/904839552
available; sequence id is 189658
2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945
2009-08-04 09:49:50,167 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732',
STARTKEY => 'http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046',
ENDKEY => '', ENCODED => 469485305, OFFLINE => true, SPLIT => true, TABLE
=>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945,
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945.
Split took 1sec
2009-08-04 09:49:50,799 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,799 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,933 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991
available; sequence id is 189657
2009-08-04 09:49:50,934 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:55,807 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
in 4sec
2009-08-04 09:54:51,308 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:51,309 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:51,486 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427/2108149192
available; sequence id is 220779
2009-08-04 09:54:51,487 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:53,885 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
in 2sec
2009-08-04 09:56:14,418 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:14,519 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:56:15,314 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:56:18,137 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu9/192.168.33.5:2222
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:43793 remote=ubuntu9/192.168.33.5:2222]
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:19,035 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:19,192 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:21,436 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:25,316 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:27,437 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:27,438 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:54934 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:27,670 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: Session Expired
at

org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:27,671 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-08-04 09:56:27,671 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-08-04 09:56:27,671 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region
Server
2009-08-04 09:56:27,674 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=8, stores=16, storefiles=16, storefileIndexSize=1,
memstoreSize=0, usedHeap=202, maxHeap=2993, blockCacheSize=97949616,
blockCacheFree=529898384, blockCacheCount=843, blockCacheHitRatio=5
2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:47976 remote=ubuntu6/192.168.33.7:2222]
2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:29,527 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-08-04 09:56:29,529 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:56:29,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-08-04 09:56:29,541 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC
Server Responder
2009-08-04 09:56:29,542 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC
Server listener on 60020
2009-08-04 09:56:29,543 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-08-04 09:56:29,575 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-08-04 09:56:29,681 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.5:60020.compactor exiting
2009-08-04 09:56:29,681 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.5:60020.majorCompactionChecker exiting
2009-08-04 09:56:29,682 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.5:60020.cacheFlusher exiting
2009-08-04 09:56:29,682 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.5:60020.logFlusher exiting
2009-08-04 09:56:29,683 INFO
org.apache.hadoop.hbase.regionserver.LogRoller:
LogRoller exiting.
2009-08-04 09:56:29,694 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
java.io.IOException: java.io.IOException: Could not complete write to file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168 by
DFSClient_1458021990
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at

sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at

sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at

org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at

org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at

org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:56:29,697 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249348520980
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:56:29,699 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com

\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:56:29,699 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.5:60020
2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closing leases
2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closed leases
2009-08-04 09:56:31,488 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x322e2b5f53a0000
2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x322e2b5f53a0000
2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x322e2b5f53a0000
2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ZooKeeper: Session:
0x322e2b5f53a0000 closed
2009-08-04 09:56:31,489 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.5:60020 exiting
2009-08-04 09:56:31,490 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-08-04 09:56:31,491 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
complete
2009-08-04 09:56:31,499 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-08-04 09:56:31,500 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-08-04 09:56:31,500 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-08-04 09:56:31,503 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222
sessionTimeout=30000
wat[email protected]
2009-08-04 09:56:31,504 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:47982 remote=ubuntu6/192.168.33.7:2222]
2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful







contents in hbase-site.xml:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ubuntu6:9000/hbase</value>
<description>The directory shared by region servers.
</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see
hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2222</value>
<description>Property from ZooKeeper's config zoo.cfg.
The port at which the clients will connect.
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ubuntu2,ubuntu3,ubuntu7,ubuntu9,ubuntu6</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,
host3.mydomain.com
".
By default this is set to localhost for local and pseudo-distributed
modes
of operation. For a fully-distributed setup, this should be set to a
full
list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in
hbase-env.sh
this is the list of servers which we will start/stop ZooKeeper on.
</description>
</property>
</configuration>
Any help?

Thanks a lot,
LvZheng

Hi Stack,
Thank you very much for your explaination.
We just adjusted the value of the property "zookeeper.session.timeout"
to 120000, and we are observing the system now.
"Are nodes running on same nodes as hbase? " --Do you mean we should
have several servers running exclusively for zk cluster? But I'm afraid that
we can not have that many servers. Any suggestion?
We don't config the zk in zoo.cfg, but in hbase-site.xml. Following is
the content in hbase-site.xml about zk.
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2222</value>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>ubuntu2,ubuntu3,ubuntu7,ubuntu9,ubuntu6</value>
</property>

<property>
<name>zookeeper.session.timeout</name>
<value>120000</value>
</property>

Thanks a lot,
LvZheng

On 8/5/09, Zheng Lv wrote:
Hello,
I adjusted the option "zookeeper.session.timeout" to 120000, and then
restarted the hbase cluster and the test program. After running normally for
14

hours, one of datanodes shut down. When I restarted the hadoop and hbase,
and checked the row count of table 'webpage', I got the result of 6625,
while the

test program log telling me there should be at least 885000. There are too
many data lost. Following is the end part of the datanode log in that
server.

2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.9:45465, dest: /192.168.33.6:50010, bytes: 1214,

op: HDFS_WRITE, cliID: DFSClient_1777493426, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-402434507207277902_27468
2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_-402434507207277902_27468 terminating
2009-08-06 04:28:32,606 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44924, bytes: 446,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,612 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44925, bytes: 277022,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,770 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-5186903983646527212_27469 src: /192.168.33.5:44941 dest:

/192.168.33.6:50010
2009-08-06 04:29:35,672 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_1888582734643135148_27447 1 Exception

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected


local=/192.168.33.6:35418 remote=/192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:853)
at java.lang.Thread.run(Thread.java:619)

2009-08-06 04:29:35,673 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_1888582734643135148_27447 terminating
2009-08-06 04:29:35,683 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_1888582734643135148_27447

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_1888582734643135148_27447 received exception

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.33.6:50010, storageID=DS-1028185837-192.168.33.6

-50010-1249268609430, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)




*************************************




And following is part of the content of test program log.

insertting 880000 webpages need 51920792 ms.
insertting 881000 webpages need 51972741 ms.
insertting 882000 webpages need 52024775 ms.
09/08/06 04:32:20 WARN zookeeper.ClientCnxn: Exception closing session
0x222e91bb6b90002 to [email protected]
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu3/192.168.33.8:2222
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected local=/192.168.33.7:52496

remote=ubuntu3/192.168.
33.8:2222]
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Server connection successful
insertting 883000 webpages need 52246380 ms.
insertting 884000 webpages need 52298370 ms.
insertting 885000 webpages need 52380479 ms.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504668723_885781',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504754735_885782',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
.
.
.
.
.
.
.



Any suggestion?
Thanks a lot,
LvZheng

2009/8/5 Zheng Lv <[email protected]>

Hello,
I adjusted the option "zookeeper.session.timeout" to 120000, and then
restarted the hbase cluster and the test program. After running normally for
14

hours, one of datanodes shut down. When I restarted the hadoop and hbase,
and checked the row count of table 'webpage', I got the result of 6625,
while the

test program log telling me there should be at least 885000. There are too
many data lost. Following is the end part of the datanode log in that
server.

2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.9:45465, dest: /192.168.33.6:50010, bytes: 1214,

op: HDFS_WRITE, cliID: DFSClient_1777493426, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-402434507207277902_27468
2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_-402434507207277902_27468 terminating
2009-08-06 04:28:32,606 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44924, bytes: 446,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,612 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44925, bytes: 277022,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,770 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-5186903983646527212_27469 src: /192.168.33.5:44941 dest:

/192.168.33.6:50010
2009-08-06 04:29:35,672 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_1888582734643135148_27447 1 Exception

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected


local=/192.168.33.6:35418 remote=/192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:853)
at java.lang.Thread.run(Thread.java:619)

2009-08-06 04:29:35,673 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_1888582734643135148_27447 terminating
2009-08-06 04:29:35,683 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_1888582734643135148_27447

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_1888582734643135148_27447 received exception

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.33.6:50010, storageID=DS-1028185837-192.168.33.6

-50010-1249268609430, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)




*************************************




And following is part of the content of test program log.

insertting 880000 webpages need 51920792 ms.
insertting 881000 webpages need 51972741 ms.
insertting 882000 webpages need 52024775 ms.
09/08/06 04:32:20 WARN zookeeper.ClientCnxn: Exception closing session
0x222e91bb6b90002 to [email protected]
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu3/192.168.33.8:2222
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected local=/192.168.33.7:52496

remote=ubuntu3/192.168.
33.8:2222]
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Server connection successful
insertting 883000 webpages need 52246380 ms.
insertting 884000 webpages need 52298370 ms.
insertting 885000 webpages need 52380479 ms.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504668723_885781',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504754735_885782',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
.
.
.
.
.
.
.



Any suggestion?
Thanks a lot,
LvZheng

2009/8/5 Zheng Lv <[email protected]>

Hello,
I adjusted the option "zookeeper.session.timeout" to 120000, and then
restarted the hbase cluster and the test program. After running normally for
14

hours, one of datanodes shut down. When I restarted the hadoop and hbase,
and checked the row count of table 'webpage', I got the result of 6625,
while the

test program log telling me there should be at least 885000. There are too
many data lost. Following is the end part of the datanode log in that
server.

2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.9:45465, dest: /192.168.33.6:50010, bytes: 1214,

op: HDFS_WRITE, cliID: DFSClient_1777493426, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-402434507207277902_27468
2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_-402434507207277902_27468 terminating
2009-08-06 04:28:32,606 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44924, bytes: 446,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,612 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44925, bytes: 277022,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,770 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-5186903983646527212_27469 src: /192.168.33.5:44941 dest:

/192.168.33.6:50010
2009-08-06 04:29:35,672 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_1888582734643135148_27447 1 Exception

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected


local=/192.168.33.6:35418 remote=/192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:853)
at java.lang.Thread.run(Thread.java:619)

2009-08-06 04:29:35,673 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_1888582734643135148_27447 terminating
2009-08-06 04:29:35,683 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_1888582734643135148_27447

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_1888582734643135148_27447 received exception

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.33.6:50010, storageID=DS-1028185837-192.168.33.6

-50010-1249268609430, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)




*************************************




And following is part of the content of test program log.

insertting 880000 webpages need 51920792 ms.
insertting 881000 webpages need 51972741 ms.
insertting 882000 webpages need 52024775 ms.
09/08/06 04:32:20 WARN zookeeper.ClientCnxn: Exception closing session
0x222e91bb6b90002 to [email protected]
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Attempting connection to server
ubuntu3/192.168.33.8:2222
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected local=/192.168.33.7:52496

remote=ubuntu3/192.168.
33.8:2222]
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Server connection successful
insertting 883000 webpages need 52246380 ms.
insertting 884000 webpages need 52298370 ms.
insertting 885000 webpages need 52380479 ms.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504668723_885781',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504754735_885782',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
.
.
.
.
.
.
.



Any suggestion?
Thanks a lot,
LvZheng

2009/8/5 Zheng Lv <[email protected]>

Hello,
I adjusted the option "zookeeper.session.timeout" to 120000, and then
restarted the hbase cluster and the test program. After running normally for
14

hours, one of datanodes shut down. When I restarted the hadoop and hbase,
and checked the row count of table 'webpage', I got the result of 6625,
while the

test program log telling me there should be at least 885000. There are too
many data lost. Following is the end part of the datanode log in that
server.

2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.9:45465, dest: /192.168.33.6:50010, bytes: 1214,

op: HDFS_WRITE, cliID: DFSClient_1777493426, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-402434507207277902_27468
2009-08-06 04:28:32,214 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_-402434507207277902_27468 terminating
2009-08-06 04:28:32,606 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44924, bytes: 446,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,612 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
192.168.33.6:50010, dest: /192.168.33.5:44925, bytes: 277022,

op: HDFS_READ, cliID: DFSClient_-255011821, srvID:
DS-1028185837-192.168.33.6-50010-1249268609430, blockid:
blk_-2647720945992878390_27447
2009-08-06 04:28:32,770 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-5186903983646527212_27469 src: /192.168.33.5:44941 dest:

/192.168.33.6:50010
2009-08-06 04:29:35,672 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_1888582734643135148_27447 1 Exception

java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected


local=/192.168.33.6:35418 remote=/192.168.33.5:50010]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:853)

at java.lang.Thread.run(Thread.java:619)

2009-08-06 04:29:35,673 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block
blk_1888582734643135148_27447 terminating
2009-08-06 04:29:35,683 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock
for block blk_1888582734643135148_27447

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_1888582734643135148_27447 received exception

java.io.EOFException: while trying to read 65557 bytes
2009-08-06 04:29:35,689 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
192.168.33.6:50010, storageID=DS-1028185837-192.168.33.6

-50010-1249268609430, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:264)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:308)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:372)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:524)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)

at java.lang.Thread.run(Thread.java:619)




*************************************




And following is part of the content of test program log.

insertting 880000 webpages need 51920792 ms.
insertting 881000 webpages need 51972741 ms.
insertting 882000 webpages need 52024775 ms.
09/08/06 04:32:20 WARN zookeeper.ClientCnxn: Exception closing session
0x222e91bb6b90002 to [email protected]
java.io.IOException: TIMED OUT
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Attempting connection to
server ubuntu3/192.168.33.8:2222
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected local=/192.168.33.7:52496

remote=ubuntu3/192.168.
33.8:2222]
09/08/06 04:32:21 INFO zookeeper.ClientCnxn: Server connection successful
insertting 883000 webpages need 52246380 ms.
insertting 884000 webpages need 52298370 ms.
insertting 885000 webpages need 52380479 ms.
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504668723_885781',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server, retryOnlyOne=true, index=0, islastrow=true,
tries=9,

nu
mtries=10, i=0, listsize=1, location=address: 192.168.33.5:60020,
regioninfo: REGION => {NAME => 'webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01

\x2F5FO
O155J0001124J.html1249504151762_879696,1249504267420', STARTKEY =>
'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696
', ENDKEY => '', ENCODED => 1607113409, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL =>

'2147
483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>

'2147483
647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}},
region=webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.h
tml1249504151762_879696,1249504267420 for region webpage,http:\x2F\
x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01

\x2F5FOO155J0001124J.html1249504151762_879696,1249504267
420, row 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249504754735_885782',
but failed after 10 attempts.
Exceptions:

at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1041)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
at hbasetest.HBaseWebpage.insert(HBaseWebpage.java:82)
at hbasetest.InsertThread.run(InsertThread.java:26)
.
.
.
.
.
.
.



Any suggestion?
Thanks a lot,
LvZheng

2009/8/5 Zheng Lv <[email protected]>

Hi Stack,

Hello Everyone,
We are testing hbase0.20.0 rc in a cluster with 5 nodes, inserting web
pages at a speed of 20 pages per second. A few minutes later one region
server shut down, and about 40 minutes later another shut down too, leaving
2 of 4 region servers running.
I note that the contents of logs in two servers are not similar, and
below is the contents.

ubuntu7:


2009-08-04 08:59:35,870 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249344005673,
entries=34, calcsize=10262, filesize=7353. New hlog
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858
2009-08-04 09:12:49,462 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:12:49,564 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:12:49,959 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:50860 remote=ubuntu7/192.168.33.6:2222]
2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:12:49,965 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-08-04 09:13:39,460 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:13:39,567 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:13:39,807 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:13:46,192 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu9/192.168.33.5:2222
2009-08-04 09:13:52,184 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:13:52,185 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:13:52,923 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:49662 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:13:55,342 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350000 to [email protected]
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:13:55,342 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-08-04 09:13:55,343 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-08-04 09:13:55,343 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region Server
2009-08-04 09:13:55,375 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=1, stores=2, storefiles=2, storefileIndexSize=0,
memstoreSize=0, usedHeap=24, maxHeap=2993, blockCacheSize=5151568,
blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
2009-08-04 09:13:55,833 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.6:60020.logFlusher exiting
2009-08-04 09:13:55,852 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.6:60020.cacheFlusher exiting
2009-08-04 09:13:55,882 INFO
org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-08-04 09:13:58,340 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60020
2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-08-04 09:13:58,471 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.6:60020.compactor exiting
2009-08-04 09:13:58,472 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.6:60020.majorCompactionChecker exiting
2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closing leases
2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.6:60020.leaseChecker closed leases
2009-08-04 09:14:01,895 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-08-04 09:14:25,347 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:25,348 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 26 seconds
2009-08-04 09:14:25,348 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries left
4
2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 27 seconds
2009-08-04 09:14:25,753 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries left
3
2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 28 seconds
2009-08-04 09:14:26,557 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries left
2
2009-08-04 09:14:28,161 INFO org.apache.hadoop.hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:28,162 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for
replication for 29 seconds
2009-08-04 09:14:28,162 WARN org.apache.hadoop.hdfs.DFSClient:
NotReplicatedYetException sleeping
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries left
1
2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block null bad datanode[0] nodes == null
2009-08-04 09:14:31,368 WARN org.apache.hadoop.hdfs.DFSClient: Could not
get block locations. Source file
"/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858" -
Aborting...
2009-08-04 09:14:31,371 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does
not exist. Holder DFSClient_1310853810 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:14:31,373 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed .META.,,1
2009-08-04 09:14:31,373 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.6:60020
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x22e2b69b350000
2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Session:
0x22e2b69b350000 closed
2009-08-04 09:14:31,374 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.6:60020 exiting
2009-08-04 09:14:31,377 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-08-04 09:14:31,391 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-08-04 09:14:31,392 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
2009-08-04 09:14:31,394 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-08-04 09:14:31,394 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-08-04 09:14:31,397 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222
sessionTimeout=30000
wat[email protected]
2009-08-04 09:14:31,398 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.6:49672 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:14:31,417 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: SyncConnected, type: None, path: null
2009-08-04 09:14:31,447 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2009-08-04 09:14:31,447 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:14:31,448 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:14:31,452 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that
region server is shutting down at: 192.168.33.6:60020
2009-08-04 09:14:31,453 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed to send exiting
message to master:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:660)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:14:31,453 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at:
192.168.33.6:60020
2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x122e2b7548e0087
2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x122e2b7548e0087
2009-08-04 09:14:31,460 INFO org.apache.zookeeper.ClientCnxn: Exception
while closing send thread for session 0x122e2b7548e0087 : Read error rc = -1
java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]








ubuntu9:

2009-08-04 08:59:40,655 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,,1249347647788
2009-08-04 08:59:40,656 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,,1249347647788
2009-08-04 08:59:40,683 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249347647788/491403556 available; sequence id is 0
2009-08-04 09:01:40,266 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090,
entries=3147, calcsize=29823872, filesize=29491785. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254
2009-08-04 09:01:40,269 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090 whose
highest sequence/edit id is 5984
2009-08-04 09:10:31,649 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254,
entries=6720, calcsize=63790080, filesize=63079139. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638
2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638,
entries=6715, calcsize=63794066, filesize=63090684. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324
2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254 whose
highest sequence/edit id is 12705
2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324,
entries=6710, calcsize=63797225, filesize=63093218. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979
2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638 whose
highest sequence/edit id is 19419
2009-08-04 09:12:40,827 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249347647788
2009-08-04 09:12:49,963 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249347647788 in 9sec
2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979,
entries=6720, calcsize=63800107, filesize=63095030. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060
2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324 whose
highest sequence/edit id is 26134
2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060,
entries=6715, calcsize=63798666, filesize=63094154. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426
2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979 whose
highest sequence/edit id is 32850
2009-08-04 09:14:56,116 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:14:57,075 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:44326 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:15:15,359 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249347647788
2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249347647788 in 5sec
2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,,1249347647788
2009-08-04 09:15:21,963 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249347647788
2009-08-04 09:15:24,791 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249348520980/935635885 available; sequence id is 45742
2009-08-04 09:15:24,792 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249348520980
2009-08-04 09:15:24,995 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380
available; sequence id is 45743
2009-08-04 09:15:24,996 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:25,039 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,,1249347647788', STARTKEY => '', ENDKEY => '', ENCODED =>
491403556, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'webpage',
FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2',
TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}, {NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS =>
'1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}]}}, new regions: webpage,,1249348520980,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980.
Split took 4sec
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,,1249348520980
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:26,517 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,,1249348520980
2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,,1249348520980/935635885 available; sequence id is 45742
2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,,1249348520980
2009-08-04 09:15:26,690 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:15:26,998 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380
available; sequence id is 45743
2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,,1249348520980 in 4sec
2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:16:00,015 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 0 on 60020 took 25134ms appending an edit to hlog;
editcount=6668
2009-08-04 09:16:00,016 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 19781ms optional sync'ing
hlog; editcount=6674
2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426,
entries=6710, calcsize=63779032, filesize=63075244. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377
2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060 whose
highest sequence/edit id is 39565
2009-08-04 09:16:30,001 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC
Server handler 1 on 60020 took 27826ms appending an edit to hlog;
editcount=204
2009-08-04 09:16:30,002 WARN org.apache.hadoop.hbase.regionserver.HLog:
regionserver/192.168.33.5:60020.logFlusher took 19986ms optional sync'ing
hlog; editcount=205
2009-08-04 09:16:38,451 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
2009-08-04 09:16:38,457 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020, call openScanner([[email protected], startRow=, stopRow=,
maxVersions=1, timeRange=[0,9223372036854775807),
families={(family=CF_CONTENT, columns={}), (family=CF_INFORMATION,
columns={}}) from 192.168.33.7:52275: error:
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377,
entries=6955, calcsize=66566006, filesize=65836364. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837
2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426 whose
highest sequence/edit id is 46277
2009-08-04 09:17:28,853 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377 whose
highest sequence/edit id is 53231
2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
in 1mins, 59sec
2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:17:30,218 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980
2009-08-04 09:17:33,371 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797
available; sequence id is 53366
2009-08-04 09:17:33,372 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:33,575 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105
available; sequence id is 53367
2009-08-04 09:17:33,576 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:34,580 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 0 time(s).
2009-08-04 09:17:35,581 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 1 time(s).
2009-08-04 09:17:36,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 2 time(s).
2009-08-04 09:17:37,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 3 time(s).
2009-08-04 09:17:38,583 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 4 time(s).
2009-08-04 09:17:39,584 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 5 time(s).
2009-08-04 09:17:40,585 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 6 time(s).
2009-08-04 09:17:41,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 7 time(s).
2009-08-04 09:17:42,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 8 time(s).
2009-08-04 09:17:43,587 INFO org.apache.hadoop.ipc.HBaseClient: Retrying
connect to server: /192.168.33.6:60020. Already tried 9 time(s).
2009-08-04 09:17:45,611 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744',
ENDKEY => '', ENCODED => 1870783380, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989.
Split took 15sec
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:47,754 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,943 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797
available; sequence id is 53366
2009-08-04 09:17:47,944 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:17:47,944 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:17:48,163 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105
available; sequence id is 53367
2009-08-04 09:17:52,777 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
in 4sec
2009-08-04 09:17:52,778 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:18:01,313 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
in 8sec
2009-08-04 09:24:26,812 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837,
entries=6661, calcsize=63769148, filesize=63059485. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799
2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799,
entries=6661, calcsize=63769500, filesize=63059617. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458
2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837 whose
highest sequence/edit id is 59894
2009-08-04 09:25:34,681 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
in 4sec
2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:39,244 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989
2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000/1497560335
available; sequence id is 68239
2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000
2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000/1242271099
available; sequence id is 68240
2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000
2009-08-04 09:25:42,663 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168',
ENDKEY => '', ENCODED => 203332105, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION =>
'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000.
Split took 3sec
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:24,918 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:25,058 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035/2028268999
available; sequence id is 66248
2009-08-04 09:30:25,059 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:30:25,061 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:25,299 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035/1041724375
available; sequence id is 66249
2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
in 3sec
2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:30:41,191 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
in 12sec
2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458,
entries=6662, calcsize=63774480, filesize=63064760. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432
2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799 whose
highest sequence/edit id is 66556
2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432,
entries=6661, calcsize=63776160, filesize=63066278. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174
2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458 whose
highest sequence/edit id is 73217
2009-08-04 09:32:35,692 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
in 2sec
2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:38,479 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035
2009-08-04 09:32:40,510 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326/481931133
available; sequence id is 82900
2009-08-04 09:32:42,063 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326
2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637
available; sequence id is 82901
2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:42,298 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898',
ENDKEY => '', ENCODED => 1041724375, OFFLINE => true, SPLIT => true, TABLE
=> {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326.
Split took 3sec
2009-08-04 09:32:43,169 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:43,170 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637
available; sequence id is 82901
2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
in 11sec
2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:55,912 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326
2009-08-04 09:32:56,630 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115
available; sequence id is 83905
2009-08-04 09:32:56,631 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:56,793 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028
available; sequence id is 83906
2009-08-04 09:32:56,794 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:56,808 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298',
ENDKEY => '', ENCODED => 922277637, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293.
Split took 1sec
2009-08-04 09:32:58,205 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,206 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:58,206 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,304 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028
available; sequence id is 83906
2009-08-04 09:32:58,304 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:32:58,306 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:32:58,441 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115
available; sequence id is 83905
2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
in 5sec
2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:33:09,988 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
in 6sec
2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174,
entries=6658, calcsize=63774356, filesize=63065350. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082
2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432 whose
highest sequence/edit id is 79878
2009-08-04 09:34:22,724 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082,
entries=6666, calcsize=63777964, filesize=63067534. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713
2009-08-04 09:34:22,725 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174 whose
highest sequence/edit id is 86545
2009-08-04 09:35:06,800 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
in 2sec
2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:09,561 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293
2009-08-04 09:35:12,780 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334/316888594
available; sequence id is 98604
2009-08-04 09:35:12,781 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334
2009-08-04 09:35:12,949 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334/1414734782
available; sequence id is 98605
2009-08-04 09:35:12,950 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334
2009-08-04 09:35:12,966 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720',
ENDKEY => '', ENCODED => 112267028, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334.
Split took 3sec
2009-08-04 09:37:58,723 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:37:58,724 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:38:18,205 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622/1005978655
available; sequence id is 113675
2009-08-04 09:38:18,206 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:38:22,211 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
in 4sec
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:32,557 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732/671938285
available; sequence id is 174598
2009-08-04 09:47:32,694 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:47:32,913 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732/469485305
available; sequence id is 174599
2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
in 2sec
2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:47:46,699 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
in 11sec
2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713,
entries=6662, calcsize=63776160, filesize=63066462. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966
2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082 whose
highest sequence/edit id is 93207
2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966,
entries=6661, calcsize=63776160, filesize=63066278. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537
2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713 whose
highest sequence/edit id is 175865
2009-08-04 09:49:42,602 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:44,186 INFO org.apache.hadoop.hbase.regionserver.HLog:
Roll /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537,
entries=6661, calcsize=63776160, filesize=63066258. New hlog
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168
2009-08-04 09:49:44,187 INFO org.apache.hadoop.hbase.regionserver.HLog:
removing old hlog file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966 whose
highest sequence/edit id is 182525
2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
in 6sec
2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:49,199 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732
2009-08-04 09:49:49,954 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991
available; sequence id is 189657
2009-08-04 09:49:49,955 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945/904839552
available; sequence id is 189658
2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945
2009-08-04 09:49:50,167 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META
updated, and report to master all successful. Old region=REGION => {NAME =>
'webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732',
STARTKEY => 'http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046',
ENDKEY => '', ENCODED => 469485305, OFFLINE => true, SPLIT => true, TABLE =>
{{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2',
COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945,
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945.
Split took 1sec
2009-08-04 09:49:50,799 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,799 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:50,933 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991
available; sequence id is 189657
2009-08-04 09:49:50,934 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:49:55,807 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
in 4sec
2009-08-04 09:54:51,308 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:51,309 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:51,486 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427/2108149192
available; sequence id is 220779
2009-08-04 09:54:51,487 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:54:53,885 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region webpage,http:\x2F\x2Fnews.163.com<http://x2fnews.163.com/>\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
in 2sec
2009-08-04 09:56:14,418 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:14,519 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Disconnected, type: None, path: null
2009-08-04 09:56:15,314 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:56:18,137 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu9/192.168.33.5:2222
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:43793 remote=ubuntu9/192.168.33.5:2222]
2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:19,035 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:19,192 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:21,436 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22e2b69b350001 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:25,316 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:27,437 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-04 09:56:27,438 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:54934 remote=ubuntu3/192.168.33.8:2222]
2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:27,670 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x322e2b5f53a0000 to [email protected]
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-04 09:56:27,671 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
state: Expired, type: None, path: null
2009-08-04 09:56:27,671 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
expired
2009-08-04 09:56:27,671 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region Server
2009-08-04 09:56:27,674 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=8, stores=16, storefiles=16, storefileIndexSize=1,
memstoreSize=0, usedHeap=202, maxHeap=2993, blockCacheSize=97949616,
blockCacheFree=529898384, blockCacheCount=843, blockCacheHitRatio=5
2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:47976 remote=ubuntu6/192.168.33.7:2222]
2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-04 09:56:29,527 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2009-08-04 09:56:29,529 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-08-04 09:56:29,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2009-08-04 09:56:29,541 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2009-08-04 09:56:29,542 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60020
2009-08-04 09:56:29,543 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2009-08-04 09:56:29,575 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2009-08-04 09:56:29,681 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/192.168.33.5:60020.compactor exiting
2009-08-04 09:56:29,681 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
regionserver/192.168.33.5:60020.majorCompactionChecker exiting
2009-08-04 09:56:29,682 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver/192.168.33.5:60020.cacheFlusher exiting
2009-08-04 09:56:29,682 INFO
org.apache.hadoop.hbase.regionserver.LogFlusher:
regionserver/192.168.33.5:60020.logFlusher exiting
2009-08-04 09:56:29,683 INFO
org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2009-08-04 09:56:29,694 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
abort
java.io.IOException: java.io.IOException: Could not complete write to file
/hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168 by
DFSClient_1458021990
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631)
at java.lang.Thread.run(Thread.java:619)
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989
2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035
2009-08-04 09:56:29,697 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,,1249348520980
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293
2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622
2009-08-04 09:56:29,699 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed webpage,http:\x2F\x2Fnews.163.com <http://x2fnews.163.com/>
\x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732
2009-08-04 09:56:29,699 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.33.5:60020
2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closing leases
2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.33.5:60020.leaseChecker closed leases
2009-08-04 09:56:31,488 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x322e2b5f53a0000
2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x322e2b5f53a0000
2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x322e2b5f53a0000
2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ZooKeeper: Session:
0x322e2b5f53a0000 closed
2009-08-04 09:56:31,489 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
192.168.33.5:60020 exiting
2009-08-04 09:56:31,490 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
thread.
2009-08-04 09:56:31,491 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
2009-08-04 09:56:31,499 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HRegionServer, port=60020
2009-08-04 09:56:31,500 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
2009-08-04 09:56:31,500 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
2009-08-04 09:56:31,503 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection,
host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222
sessionTimeout=30000
wat[email protected]
2009-08-04 09:56:31,504 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu6/192.168.33.7:2222
2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.5:47982 remote=ubuntu6/192.168.33.7:2222]
2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful







contents in hbase-site.xml:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ubuntu6:9000/hbase</value>
<description>The directory shared by region servers.
</description>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
<description>The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed
Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see
hbase-env.sh)
</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2222</value>
<description>Property from ZooKeeper's config zoo.cfg.
The port at which the clients will connect.
</description>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>ubuntu2,ubuntu3,ubuntu7,ubuntu9,ubuntu6</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,
host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed
modes
of operation. For a fully-distributed setup, this should be set to a
full
list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in
hbase-env.sh
this is the list of servers which we will start/stop ZooKeeper on.
</description>
</property>
</configuration>
Any help?

Thanks a lot,
LvZheng