Grokbase Groups HBase user July 2009
FAQ
Hi All,

We have a cluster with 4 nodes, one for the Master server, and others
for Region Server.

In our case, there are 3 Downloaders crawling some specific web pages
which will be saved in HBase(0.19.3) then.
After running the Downloaders for a while, we found that there are
tons of Exceptions in the hbase log files like this:

2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
of ~1.8m for region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
in 316ms, sequence id=4774267, compaction requested=true
2009-07-27 12:12:25,833 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_INFORMATION
2009-07-27 12:12:25,834 DEBUG
org.apache.hadoop.hbase.regionserver.HStore: closed
1734514356/CF_CONTENT
2009-07-27 12:12:25,834 INFO
org.apache.hadoop.hbase.regionserver.HRegion: Closed
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,837 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
org.apache.hadoop.hbase.NotServingRegionException: Region
webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
closed
at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


And from then , the Downloaders got a big drop in the speed of page crawling.

Any ideas?


--
Regards
Angus

Search Discussions

  • Ninad Raut at Jul 27, 2009 at 10:02 am
    I think there is a swap happening on regionservers due to less memory...
    increase the allotted memory in hadoop-site.xml for all the slaves.
    On Mon, Jul 27, 2009 at 3:19 PM, Angus He wrote:

    Hi All,

    We have a cluster with 4 nodes, one for the Master server, and others
    for Region Server.

    In our case, there are 3 Downloaders crawling some specific web pages
    which will be saved in HBase(0.19.3) then.
    After running the Downloaders for a while, we found that there are
    tons of Exceptions in the hbase log files like this:

    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
    of ~1.8m for region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    in 316ms, sequence id=4774267, compaction requested=true
    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_INFORMATION
    2009-07-27 12:12:25,834 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_CONTENT
    2009-07-27 12:12:25,834 INFO
    org.apache.hadoop.hbase.regionserver.HRegion: Closed
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    2009-07-27 12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
    9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
    9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


    And from then , the Downloaders got a big drop in the speed of page
    crawling.

    Any ideas?


    --
    Regards
    Angus
  • Angus He at Jul 28, 2009 at 3:58 am
    Hi Ninad,

    Thanks for your comments.

    We have already committed 3G memory to each of the region servers.
    And the following texts are excerpted from hbase-env.sh in our cluster.

    # The java implementation to use. Java 1.6 required.
    export JAVA_HOME=/usr/lib/jvm/java-6-sun

    # The maximum amount of heap to use, in MB. Default is 1000.
    export HBASE_HEAPSIZE=3000

    # Extra Java runtime options. Empty by default.
    export HBASE_OPTS="-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
    -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
    -Xloggc:/opt/hbase/logs/gc-hbase.log"


    On Mon, Jul 27, 2009 at 6:03 PM, Ninad Rautwrote:
    I think there is a swap happening on regionservers due to less memory...
    increase the allotted memory in hadoop-site.xml for all the slaves.

    On Mon, Jul 27, 2009 at 3:19 PM, Angus He wrote:


    --
    Regards
    Angus
  • Stack at Jul 28, 2009 at 3:44 am
    I was looking at this today.

    When you start to get the NotServingRegionException, has the region just
    split? You can tell by looking for "webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868" in the
    master log. If it has a message about splitting just before you get the
    below NSRE exception, then it seems like its a case of HBASE-1671.
    Hopefully will have a fix this evening.

    St.Ack
    On Mon, Jul 27, 2009 at 2:49 AM, Angus He wrote:

    Hi All,

    We have a cluster with 4 nodes, one for the Master server, and others
    for Region Server.

    In our case, there are 3 Downloaders crawling some specific web pages
    which will be saved in HBase(0.19.3) then.
    After running the Downloaders for a while, we found that there are
    tons of Exceptions in the hbase log files like this:

    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
    of ~1.8m for region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    in 316ms, sequence id=4774267, compaction requested=true
    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_INFORMATION
    2009-07-27 12:12:25,834 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_CONTENT
    2009-07-27 12:12:25,834 INFO
    org.apache.hadoop.hbase.regionserver.HRegion: Closed
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
    9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
    9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


    And from then , the Downloaders got a big drop in the speed of page
    crawling.

    Any ideas?


    --
    Regards
    Angus
  • Angus He at Jul 28, 2009 at 4:21 am
    Yup , I found this in the Master log file.

    2009-07-27 12:13:10,880 INFO
    org.apache.hadoop.hbase.master.ServerManager: Received
    MSG_REPORT_SPLIT:
    webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868:
    webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    split; daughters:
    webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248667945476,
    webpage,http://www.tianya.cn/techforum/content/50/565730.shtml,1248667945476:
    safeMode=false from 192.168.33.5:60020

    BTW, There is about a 50-second time lag between Region server and
    Master server.


    On Tue, Jul 28, 2009 at 11:45 AM, stackwrote:
    I was looking at this today.

    When you start to get the NotServingRegionException, has the region just
    split?  You can tell by looking for "webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868" in the
    master log.  If it has a message about splitting just before you get the
    below NSRE exception, then it seems like its a case of HBASE-1671.
    Hopefully will have a fix this evening.

    St.Ack
    On Mon, Jul 27, 2009 at 2:49 AM, Angus He wrote:

    Hi All,

    We have a cluster with 4 nodes, one for the Master server, and others
    for Region Server.

    In our case, there are 3 Downloaders crawling some specific web pages
    which will be saved in HBase(0.19.3) then.
    After running the Downloaders for a while,  we found that there are
    tons of Exceptions in the hbase log files like this:

    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush
    of ~1.8m for region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    in 316ms, sequence id=4774267, compaction requested=true
    2009-07-27 12:12:25,833 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_INFORMATION
    2009-07-27 12:12:25,834 DEBUG
    org.apache.hadoop.hbase.regionserver.HStore: closed
    1734514356/CF_CONTENT
    2009-07-27 12:12:25,834 INFO
    org.apache.hadoop.hbase.regionserver.HRegion: Closed
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,837 ERROR
    org.apache.hadoop.hbase.regionserver.HRegionServer:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 2 on 60020, call getRow([B@187885c2, [B@1095901b, null,
    9223372036854775807, 1, -1) from 192.168.33.9:59836: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
    2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC
    Server handler 5 on 60020, call getRow([B@28333b1e, [B@3b6e4330, null,
    9223372036854775807, 1, -1) from 192.168.33.5:58297: error:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    org.apache.hadoop.hbase.NotServingRegionException: Region
    webpage,
    http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)


    And from then ,  the Downloaders got a big drop in the speed of page
    crawling.

    Any ideas?


    --
    Regards
    Angus


    --
    Regards
    Angus

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedJul 27, '09 at 9:48a
activeJul 28, '09 at 4:21a
posts5
users3
websitehbase.apache.org

3 users in discussion

Angus He: 3 posts Ninad Raut: 1 post Stack: 1 post

People

Translate

site design / logo © 2022 Grokbase