FAQ
Hi ,

I run map/reduce job which from reducers writes to hbase. I have to write
8-9 million raws.

My environment:
5 servers dual core , 8G ram.
HDFS -- 9TB

hadoop has 1000 heap_size ,
hbase has 2000 heap_size
zookeeper has 1000 heap_size
20 map / 16 reducers while running the job

Allmost all the time get such exception

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337 closed
at
org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1607)
at
org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1731)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)



Thanks in advance.

Search Discussions

  • Jonathan Gray at Nov 8, 2010 at 6:55 pm
    NSRE is normal, this happens when regions move around and your client needs to update the location.

    That seems like an awful lot of mappers/reducers on a 5 server / dual core setup... You have only 2 cores per server but you have a DataNode, RegionServer, and 4 map tasks and 3 reduce tasks? That may be over extending things a bit.

    JG
    -----Original Message-----
    From: Oleg Ruchovets
    Sent: Monday, November 08, 2010 7:52 AM
    To: user@hbase.apache.org
    Subject: Hbase insertion process cause to region server down.

    Hi ,

    I run map/reduce job which from reducers writes to hbase. I have to
    write
    8-9 million raws.

    My environment:
    5 servers dual core , 8G ram.
    HDFS -- 9TB

    hadoop has 1000 heap_size ,
    hbase has 2000 heap_size
    zookeeper has 1000 heap_size
    20 map / 16 reducers while running the job

    Allmost all the time get such exception

    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
    20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
    :1607)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
    va:1731)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
    rImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
    5)



    Thanks in advance.
  • Hari Sreekumar at Nov 9, 2010 at 4:26 am
    Hi Jonathan,

    I have faced the exact same issue as Oleg and it seems to
    happen randomly. Sometimes I get this exception and the task fails,
    sometimes I don't and it succeeds. Do you mean that this is normal behavior?

    hari
    On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray wrote:

    NSRE is normal, this happens when regions move around and your client needs
    to update the location.

    That seems like an awful lot of mappers/reducers on a 5 server / dual core
    setup... You have only 2 cores per server but you have a DataNode,
    RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
    extending things a bit.

    JG
    -----Original Message-----
    From: Oleg Ruchovets
    Sent: Monday, November 08, 2010 7:52 AM
    To: user@hbase.apache.org
    Subject: Hbase insertion process cause to region server down.

    Hi ,

    I run map/reduce job which from reducers writes to hbase. I have to
    write
    8-9 million raws.

    My environment:
    5 servers dual core , 8G ram.
    HDFS -- 9TB

    hadoop has 1000 heap_size ,
    hbase has 2000 heap_size
    zookeeper has 1000 heap_size
    20 map / 16 reducers while running the job

    Allmost all the time get such exception

    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
    20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
    :1607)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
    va:1731)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
    rImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
    5)



    Thanks in advance.
  • Oleg Ruchovets at Nov 9, 2010 at 9:20 am
    As Jonathan said it is normal , I still need to do more tests to be sure
    that I hasn't lost the inserted data.

    I paid attention that I got this exceptions during compaction

    2010-11-08 18:28:53,663 DEBUG
    org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
    started. Attempting to free 52292232 bytes
    2010-11-08 18:28:53,666 DEBUG
    org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction
    completed. Freed 52309408 bytes. Priority Sizes: Single=57.505157MB

    2010-11-08 18:29:22,111 INFO org.apache.hadoop.hbase.regionserver.HRegion:
    Starting compaction on region
    URLs_sanity,20101107_n/gs/34468.html,1289233112752
    2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
    Compaction size of globalcounters: 65.5m; Skipped 1 file(s), size: 41262334
    2010-11-08 18:29:22,141 DEBUG org.apache.hadoop.hbase.regionserver.Store:
    Started compaction of 3 file(s) into
    /hbase/URLs_sanity/compaction.dir/613365657, seqid=2855632301
    2010-11-08 18:29:22,896 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
    URLs_sanity,20101107_gallery.mobile9.com/f/1139457/,1289233750914
    2010-11-08 18:29:22,897 INFO
    org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
    URLs_sanity,20101107_gallery/1139457/,1289233750914
    2010-11-08 18:29:22,897 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
    Opening region URLs_sanity,20101107_bl/1139457/,1289233750914,
    encoded=1345660410


    My questions are:
    1) what happens with insertion process when the job got NSRE exception?
    2) Could compaction cause to additional load to the system? and if yes
    what is the recommendation regarding
    compaction parameters? In my case reducers inserts to hbase almost 3 hours.
    3) In addition such behavior I got only in one machine , but I have 5.


    Oleg.

    On Tue, Nov 9, 2010 at 6:26 AM, Hari Sreekumar wrote:

    Hi Jonathan,

    I have faced the exact same issue as Oleg and it seems to
    happen randomly. Sometimes I get this exception and the task fails,
    sometimes I don't and it succeeds. Do you mean that this is normal
    behavior?

    hari
    On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray wrote:

    NSRE is normal, this happens when regions move around and your client needs
    to update the location.

    That seems like an awful lot of mappers/reducers on a 5 server / dual core
    setup... You have only 2 cores per server but you have a DataNode,
    RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
    extending things a bit.

    JG
    -----Original Message-----
    From: Oleg Ruchovets
    Sent: Monday, November 08, 2010 7:52 AM
    To: user@hbase.apache.org
    Subject: Hbase insertion process cause to region server down.

    Hi ,

    I run map/reduce job which from reducers writes to hbase. I have to
    write
    8-9 million raws.

    My environment:
    5 servers dual core , 8G ram.
    HDFS -- 9TB

    hadoop has 1000 heap_size ,
    hbase has 2000 heap_size
    zookeeper has 1000 heap_size
    20 map / 16 reducers while running the job

    Allmost all the time get such exception

    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: Region URLs_sanity,
    20101106_www.homemade-chinese-soups.com/conch-soup.html,1289163460337
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
    :1607)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
    va:1731)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
    rImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
    5)



    Thanks in advance.
  • Jonathan Gray at Nov 9, 2010 at 8:30 pm
    Yes, it is normal and expected to get some NSREs if your regions are splitting or there is rebalancing going on.

    They should not fail the task unless for some reason the client is never able to locate the region. If this is the case, something is wrong with the cluster.

    JG
    -----Original Message-----
    From: Hari Sreekumar
    Sent: Monday, November 08, 2010 8:26 PM
    To: user@hbase.apache.org
    Subject: Re: Hbase insertion process cause to region server down.

    Hi Jonathan,

    I have faced the exact same issue as Oleg and it seems to
    happen randomly. Sometimes I get this exception and the task fails,
    sometimes I don't and it succeeds. Do you mean that this is normal
    behavior?

    hari
    On Tue, Nov 9, 2010 at 12:25 AM, Jonathan Gray wrote:

    NSRE is normal, this happens when regions move around and your client needs
    to update the location.

    That seems like an awful lot of mappers/reducers on a 5 server / dual core
    setup... You have only 2 cores per server but you have a DataNode,
    RegionServer, and 4 map tasks and 3 reduce tasks? That may be over
    extending things a bit.

    JG
    -----Original Message-----
    From: Oleg Ruchovets
    Sent: Monday, November 08, 2010 7:52 AM
    To: user@hbase.apache.org
    Subject: Hbase insertion process cause to region server down.

    Hi ,

    I run map/reduce job which from reducers writes to hbase. I have to
    write
    8-9 million raws.

    My environment:
    5 servers dual core , 8G ram.
    HDFS -- 9TB

    hadoop has 1000 heap_size ,
    hbase has 2000 heap_size
    zookeeper has 1000 heap_size
    20 map / 16 reducers while running the job

    Allmost all the time get such exception

    org.apache.hadoop.hbase.NotServingRegionException:
    org.apache.hadoop.hbase.NotServingRegionException: Region
    URLs_sanity,
    20101106_www.homemade-chinese-soups.com/conch-
    soup.html,1289163460337
    closed
    at
    org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java
    :1607)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1671)
    at
    org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2380)
    at
    org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.ja
    va:1731)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
    rImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:91
    5)



    Thanks in advance.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedNov 8, '10 at 3:52p
activeNov 9, '10 at 8:30p
posts5
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase