FAQ
I have a periodic process that bulk incremental loads a set of files each time into my db. The last few runs have been resulting in bulk load failures complaining of RetriesExhausted. (I am running the last release of 0.89)



Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server b5120229.yst.yahoo.net:60020 for region vidhyash_test,r:com#mop#lady!/beauty/hair/2010-07-11/136898.shtml!http,1292936192308.7f7e7521764636e108de079799ad9e44., row 'r:com#mop#lady!/star/2010-06-10/131380.shtml!http', but failed after 10 attempts.




I looked into the logs of the particular regionserver and I noticed that one of IPC handlers complains of an output error and throws an exception and after that, all it does is just validate hfiles whenever there is an attempt at a bulk incremental load. And, it isnt accessible even through the web interface.. But the region server is still alive according to the master/zk. Can you let me know what the problem is? (Below is the log where the problem arose).





2010-12-22 21:31:57,679 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 for inclusion in store metadata region vidhyash_test,r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!http,1292936187861.9ad1eccc9cf7f82282757e2b82c45559.2010-12-22 21:31:57,680 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8607417804107886121_8839017 from any node: java.io.IOException: No live nodes contain current block
2010-12-22 21:31:59,610 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=19.67 MB, free=2.32 GB, max=2.34 GB, blocks=0, accesses=1654363, hits=0, hitRatio=0.00%%, evictions=0, evicted=0, evictedPerRun=NaN2010-12-22 21:32:00,684 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8607417804107886121_8839017 from any node: java.io.IOException: No
live nodes contain current block2010-12-22 21:32:03,687 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8607417804107886121_8839017 from any node: java.io.IOException: No live nodes contain current block
2010-12-22 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: HFile bounds: first=r:jp#co#yahoo#auctions#page17#www!/jp/auction/v12791536!http last=r:jp#co#yahoo#auctions#page19!/jp/show/discussion?aID=x120219484&u=chikyuud!http2010-12-22 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: Region bounds: first=r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!h
ttp last=r:jp#co#yahoo#auctions#page19!/jp/show/reviews?aID=x144625371!http2010-12-22 21:32:06,691 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming bulk load file /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 to hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876
2010-12-22 21:32:06,695 INFO org.apache.hadoop.hbase.regionserver.Store: Moved hfile /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632
567128036272117 into store directory hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata - updating store file list.2010-12-22 21:32:06,695 INFO org.apache.hadoop.hbase.regionserver.Store: Successfully loaded store file /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117 into store metadata (new location: hdfs://b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876)2010-12-22 21:32:06,695 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call bulkLoadHFile(/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117, [B@2b6105a8, [B@6eba6ed7) from 74.6.71.45:52379: output error2010-12-22 21:32:06,696 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 on 60020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1224)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:708)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:773)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1035)
2010-12-22 21:35:33,312 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300 for inclusion in store content region vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
2010-12-22 21:35:41,324 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300 for inclusion in store content region vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
2010-12-22 21:35:54,128 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/291138288447336298 for inclusion in store content region vidhyash_test,r:la#net#kpl#www!/english/news/edn13.htm!http,1292936187014.695817b0e3a8c894240668db0448f8bf.
2010-12-22 21:35:54,529 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2731042733857819920 for inclusion in store metadata region vidhyash_test,r:com#homebargear#www!/irish-gift-set.html!http,1292936193313.99fa66ab17756ce4ce5ba3a0d8ee8799.
2010-12-22 21:35:54,823 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1898525064358474151 for inclusion in store metadata region vidhyash_test,r:fr#dazibaoueb#www!/tag.php?tag=DESINFORMATION!http,1292936188821.85b144d3f968029a903506bdb4e60cf7.
2010-12-22 21:35:55,198 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2942947977847604920 for inclusion in store content region vidhyash_test,r:com#pld#mosc95#www!/projects02/ww2/germantanks.html!http,1292936191711.8e3d5df4c05c60f674a7b78474f83eea.
2010-12-22 21:35:57,299 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2776824162022751481 for inclusion in store metadata region vidhyash_test,r:cn#com#sina#news!/c/2006-07-12/192710404866.shtml!http,1292936195947.b3b27d1cc94a6378ab4da90acad4efbf.
2010-12-22 21:35:57,547 INFO org.apache.hadoop.hbase.regionserver.Store: Validating hfile at /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1753280038544504583 for inclusion in store metadata region vidhyash_test,r:com#yoka#space!/blog/34726!http,1292936189782.714fc4e266abca11f578fd90a3561337.

Search Discussions

  • Todd Lipcon at Dec 23, 2010 at 8:32 pm
    Hey Vidhya,

    Can you get a jstack of the frozen server?

    -Todd
    On Thu, Dec 23, 2010 at 4:55 AM, Vidhyashankar Venkataraman wrote:


    I have a periodic process that bulk incremental loads a set of files each
    time into my db. The last few runs have been resulting in bulk load failures
    complaining of RetriesExhausted. (I am running the last release of 0.89)



    Exception in thread "main"
    org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
    region server b5120229.yst.yahoo.net:60020 for region
    vidhyash_test,r:com#mop#lady!/beauty/hair/2010-07-11/136898.shtml!http,1292936192308.7f7e7521764636e108de079799ad9e44.,
    row 'r:com#mop#lady!/star/2010-06-10/131380.shtml!http', but failed after 10
    attempts.




    I looked into the logs of the particular regionserver and I noticed that
    one of IPC handlers complains of an output error and throws an exception and
    after that, all it does is just validate hfiles whenever there is an attempt
    at a bulk incremental load. And, it isnt accessible even through the web
    interface.. But the region server is still alive according to the master/zk.
    Can you let me know what the problem is? (Below is the log where the problem
    arose).





    2010-12-22 21:31:57,679 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
    for inclusion in store metadata region
    vidhyash_test,r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!http,1292936187861.9ad1eccc9cf7f82282757e2b82c45559.2010-12-22
    21:31:57,680 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block
    blk_8607417804107886121_8839017 from any node: java.io.IOException: No live
    nodes contain current block
    2010-12-22 21:31:59,610 DEBUG
    org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=19.67 MB,
    free=2.32 GB, max=2.34 GB, blocks=0, accesses=1654363, hits=0,
    hitRatio=0.00%%, evictions=0, evicted=0, evictedPerRun=NaN2010-12-22
    21:32:00,684 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block
    blk_8607417804107886121_8839017 from any node: java.io.IOException: No
    live nodes contain current block2010-12-22 21:32:03,687 INFO
    org.apache.hadoop.hdfs.DFSClient: Could not obtain block
    blk_8607417804107886121_8839017 from any node: java.io.IOException: No live
    nodes contain current block
    2010-12-22 21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store:
    HFile bounds:
    first=r:jp#co#yahoo#auctions#page17#www!/jp/auction/v12791536!http
    last=r:jp#co#yahoo#auctions#page19!/jp/show/discussion?aID=x120219484&u=chikyuud!http2010-12-22
    21:32:06,691 DEBUG org.apache.hadoop.hbase.regionserver.Store: Region
    bounds: first=r:jp#co#yahoo#auctions#page17!/jp/show/reviews?aID=v54204554!h
    ttp
    last=r:jp#co#yahoo#auctions#page19!/jp/show/reviews?aID=x144625371!http2010-12-22
    21:32:06,691 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming bulk
    load file
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
    to hdfs://
    b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876
    2010-12-22 21:32:06,695 INFO org.apache.hadoop.hbase.regionserver.Store:
    Moved hfile
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632
    567128036272117 into store directory hdfs://
    b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata- updating store file list.2010-12-22 21:32:06,695 INFO
    org.apache.hadoop.hbase.regionserver.Store: Successfully loaded store file
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117
    into store metadata (new location: hdfs://
    b5120202.yst.yahoo.net:4600/hbase/vidhyash_test/9ad1eccc9cf7f82282757e2b82c45559/metadata/643287123673932876)2010-12-2221:32:06,695 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder,
    call
    bulkLoadHFile(/user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/3632567128036272117,
    [B@2b6105a8, [B@6eba6ed7) from 74.6.71.45:52379: output error2010-12-22
    21:32:06,696 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27
    on 60020 caught: java.nio.channels.ClosedChannelException
    at
    sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1224)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:708)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:773)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1035)
    2010-12-22 21:35:33,312 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300
    for inclusion in store content region
    vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
    2010-12-22 21:35:41,324 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2820295752884341300
    for inclusion in store content region
    vidhyash_test,r:com#careerbuilder#engineering!/en.ic/Texas_Senior-Engineer.htm!http,1292936194810.0658e436cc625b2c786ef80a5dbe4203.
    2010-12-22 21:35:54,128 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/291138288447336298
    for inclusion in store content region
    vidhyash_test,r:la#net#kpl#www!/english/news/edn13.htm!http,1292936187014.695817b0e3a8c894240668db0448f8bf.
    2010-12-22 21:35:54,529 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2731042733857819920
    for inclusion in store metadata region
    vidhyash_test,r:com#homebargear#www!/irish-gift-set.html!http,1292936193313.99fa66ab17756ce4ce5ba3a0d8ee8799.
    2010-12-22 21:35:54,823 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1898525064358474151
    for inclusion in store metadata region
    vidhyash_test,r:fr#dazibaoueb#www!/tag.php?tag=DESINFORMATION!http,1292936188821.85b144d3f968029a903506bdb4e60cf7.
    2010-12-22 21:35:55,198 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/content/2942947977847604920
    for inclusion in store content region
    vidhyash_test,r:com#pld#mosc95#www!/projects02/ww2/germantanks.html!http,1292936191711.8e3d5df4c05c60f674a7b78474f83eea.
    2010-12-22 21:35:57,299 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/2776824162022751481
    for inclusion in store metadata region
    vidhyash_test,r:cn#com#sina#news!/c/2006-07-12/192710404866.shtml!http,1292936195947.b3b27d1cc94a6378ab4da90acad4efbf.
    2010-12-22 21:35:57,547 INFO org.apache.hadoop.hbase.regionserver.Store:
    Validating hfile at
    /user/vidhyash/wcc1/debug/Ingestor/workspace/ingest_output/metadata/1753280038544504583
    for inclusion in store metadata region
    vidhyash_test,r:com#yoka#space!/blog/34726!http,1292936189782.714fc4e266abca11f578fd90a3561337.

    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedDec 23, '10 at 12:57p
activeDec 23, '10 at 8:32p
posts2
users2
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase