Grokbase Groups HBase user June 2016
FAQ
Hi,

I'm using HBase 0.94-23 with Hadoop 2.7.2. In my hadoop namenode log file,
I'm getting below log info message over and over again. No jobs are running
still seeing the same log appearing. When I run a hadoop job, some jobs
failed with timeout. The cluster has 10 data nodes with 200TB of data.
When I stop HBase, logs no longer appear.

16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 10.10.2.1:44949 is added to
blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:10.10.2.6:44949|RBW],
ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:10.10.2.1:44949|RBW],
ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:10.10.2.3:44949|RBW]]}
size 0
16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: 10.10.2.6:44949 is added to
blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:10.10.2.6:44949|RBW],
ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:10.10.2.1:44949|RBW],
ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:10.10.2.3:44949|RBW]]}
size 0
16/06/18 11:23:51 INFO hdfs.StateChange: BLOCK* allocate
blk_1077143674_3402999{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-8af58951-0eb7-4244-b76a-9356ad832a93:NORMAL:10.10.2.6:44949|RBW],
ReplicaUC[[DISK]DS-6095b306-487e-48d7-948e-84ac5ef16aa4:NORMAL:10.10.2.9:44949|RBW],
ReplicaUC[[DISK]DS-9b2f64f6-bbac-4ed5-a1f9-9366129d196b:NORMAL:10.10.2.1:44949|RBW]]}
for
/hbase/tweetTable-2016-02/d654d01588e8a46d7050852978f8eaf9/.tmp/14bc58b73bd74cf28cf7e43b2304e4d6

Appreciate any insight on this.

Regards,
Chathuri

Search Discussions

  • Ted Yu at Jun 18, 2016 at 3:37 pm
    Can you find out which region server hostsed d654d01588e8a46d7050852978f8eaf9
    and examine its log to see if there was some clue ?

    Thanks
    On Sat, Jun 18, 2016 at 8:29 AM, Chathuri Wimalasena wrote:

    Hi,

    I'm using HBase 0.94-23 with Hadoop 2.7.2. In my hadoop namenode log file,
    I'm getting below log info message over and over again. No jobs are running
    still seeing the same log appearing. When I run a hadoop job, some jobs
    failed with timeout. The cluster has 10 data nodes with 200TB of data.
    When I stop HBase, logs no longer appear.

    16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
    updated: 10.10.2.1:44949 is added to
    blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:10.10.2.1:44949
    RBW],
    ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:10.10.2.3:44949
    RBW]]}
    size 0
    16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
    updated: 10.10.2.6:44949 is added to
    blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:10.10.2.1:44949
    RBW],
    ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:10.10.2.3:44949
    RBW]]}
    size 0
    16/06/18 11:23:51 INFO hdfs.StateChange: BLOCK* allocate
    blk_1077143674_3402999{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-8af58951-0eb7-4244-b76a-9356ad832a93:NORMAL:10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-6095b306-487e-48d7-948e-84ac5ef16aa4:NORMAL:10.10.2.9:44949
    RBW],
    ReplicaUC[[DISK]DS-9b2f64f6-bbac-4ed5-a1f9-9366129d196b:NORMAL:10.10.2.1:44949
    RBW]]}
    for

    /hbase/tweetTable-2016-02/d654d01588e8a46d7050852978f8eaf9/.tmp/14bc58b73bd74cf28cf7e43b2304e4d6

    Appreciate any insight on this.

    Regards,
    Chathuri
  • Chathuri Wimalasena at Jun 18, 2016 at 5:14 pm
    Hi Ted,

    Similar error appears for different blocks. When I check the region-server
    logs for one of such region server, I can see a log like this. I have no
    idea what this means. Similar log appears in each region server logs. I
    restarted the server last night during this time and these logs showed
    after restart.

    2016-06-17 20:12:08,234 WARN org.apache.hadoop.ipc.HBaseServer:
    (responseTooSlow): {"processingtimems":13826,"call":"openRegions([{NAME =>
    'dailyTweetCountTable,,1463586976452.5bb7e255c84055fc22777aa5669d84e5.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    5bb7e255c84055fc22777aa5669d84e5,}, {NAME =>
    'geoIndexTable-2010-05,,1448647271977.ac9197ecfba5ee2079ff48603f847050.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    ac9197ecfba5ee2079ff48603f847050,}, {NAME =>
    'geoIndexTable-2010-06,,1448647263331.5ae58506ae72a7505fc8a40921b9ca01.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    5ae58506ae72a7505fc8a40921b9ca01,}, {NAME =>
    'geoIndexTable-2010-07,,1448647255029.8648eee2b4225a2ea98c8ea61a0a44a0.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    8648eee2b4225a2ea98c8ea61a0a44a0,}, {NAME =>
    'geoIndexTable-2010-08,,1448647246755.c8ba053fc36f27a9ac2c178679f0488e.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    c8ba053fc36f27a9ac2c178679f0488e,}, {NAME =>
    'geoIndexTable-2010-09,,1448647238153.d8fe74e350cacf53fc03a89900c68470.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    d8fe74e350cacf53fc03a89900c68470,}, {NAME =>
    'geoIndexTable-2010-10,,1448647229607.66137956b612a13c4836533f8bf7901b.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    66137956b612a13c4836533f8bf7901b,}, {NAME =>
    'geoIndexTable-2010-11,,1448647216117.edb26d17ad97461787129a08798b73f9.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    edb26d17ad97461787129a08798b73f9,}, {NAME =>
    'geoIndexTable-2010-12,,1448647206835.f491a9d545f0071b7d9a7523bcc0eb0d.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    f491a9d545f0071b7d9a7523bcc0eb0d,}, {NAME =>
    'geoIndexTable-2011-01,,1448334295446.9b5b7fb57f2ba5b964596cb9d5e7dc53.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    9b5b7fb57f2ba5b964596cb9d5e7dc53,}, {NAME =>
    'geoIndexTable-2011-02,,1448334286105.f63660b053d21c44c570e0ea0b40fda9.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    f63660b053d21c44c570e0ea0b40fda9,}, {NAME =>
    'geoIndexTable-2011-03,,1448334276625.15ca34fb9bf26742a81bb3059903df85.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    15ca34fb9bf26742a81bb3059903df85,}, {NAME =>
    'geoIndexTable-2011-04,,1448334253107.46c25193de28f3729eb5c82b02d57d1e.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    46c25193de28f3729eb5c82b02d57d1e,}, {NAME =>
    'geoIndexTable-2011-05,,1448334241005.e1b70b296cd1b6d458b46fde8562b99e.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    e1b70b296cd1b6d458b46fde8562b99e,}, {NAME =>
    'geoIndexTable-2011-06,,1448334231051.6b0624afcd76d17b8bd89d75343486e2.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    6b0624afcd76d17b8bd89d75343486e2,}, {NAME =>
    'geoIndexTable-2011-07,,1448334221528.a44521ed13506716f9c70e774596137a.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    a44521ed13506716f9c70e774596137a,}, {NAME =>
    'geoIndexTable-2011-08,,1448334211917.a2051bbac9578fabf21bfb071c7fcfd3.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    a2051bbac9578fabf21bfb071c7fcfd3,}, {NAME =>
    'geoIndexTable-2011-09,,1448334201977.d1ec93239fdc616911da9a3802907462.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    d1ec93239fdc616911da9a3802907462,}, {NAME =>
    'geoIndexTable-2011-10,,1448334189415.969b69327d39e2e27d27afbbea867ebb.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    969b69327d39e2e27d27afbbea867ebb,}, {NAME =>
    'geoIndexTable-2011-11,,1448334179915.fa2093ff09eeb2b80728e28c3cc6dc52.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    fa2093ff09eeb2b80728e28c3cc6dc52,}, {NAME =>
    'geoIndexTable-2011-12,,1448334170540.1c8da5e32729c3d950a86f2b4c77b1f3.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    1c8da5e32729c3d950a86f2b4c77b1f3,}, {NAME =>
    'geoIndexTable-2012-01,,1447981747677.e798784dac6d307db89b924a93239bae.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    e798784dac6d307db89b924a93239bae,}, {NAME =>
    'geoIndexTable-2012-02,,1447981738161.6f3a5c1ba77bb4cab7fcc7298e566e6b.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    6f3a5c1ba77bb4cab7fcc7298e566e6b,}, {NAME =>
    'geoIndexTable-2012-03,,1447981728637.f92fa7dda5ad4b6910cdd8575ecba3f2.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    f92fa7dda5ad4b6910cdd8575ecba3f2,}, {NAME =>
    'geoIndexTable-2012-04,,1447981718427.5aca6cedd937eb8fa7ca667652ec363a.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    5aca6cedd937eb8fa7ca667652ec363a,}, {NAME =>
    'geoIndexTable-2012-05,,1447981709307.4043f99b2c2b1da50dc9c5366b52550f.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    4043f99b2c2b1da50dc9c5366b52550f,}, {NAME =>
    'geoIndexTable-2012-06,,1447981700265.fb6694ced4c3662abe48c2c12d7b31e3.',
    STARTKEY => '', ENDKEY => '', ENCODED =>
    fb6694ced4c3662abe48c2c12d7b31e3,}, {NAME =>
    'geoIndexTable-2012-07,,1447981689467.32072590c0d6ebf70a264e4ca0f1f28a.',
    STARTKEY => '', ENDKEY => '', E
    On Sat, Jun 18, 2016 at 11:37 AM, Ted Yu wrote:

    Can you find out which region server hostsed
    d654d01588e8a46d7050852978f8eaf9
    and examine its log to see if there was some clue ?

    Thanks

    On Sat, Jun 18, 2016 at 8:29 AM, Chathuri Wimalasena <kamalasini@gmail.com
      wrote:
    Hi,

    I'm using HBase 0.94-23 with Hadoop 2.7.2. In my hadoop namenode log file,
    I'm getting below log info message over and over again. No jobs are running
    still seeing the same log appearing. When I run a hadoop job, some jobs
    failed with timeout. The cluster has 10 data nodes with 200TB of data.
    When I stop HBase, logs no longer appear.

    16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
    updated: 10.10.2.1:44949 is added to
    blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:
    10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:
    10.10.2.1:44949
    RBW],
    ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:
    10.10.2.3:44949
    RBW]]}
    size 0
    16/06/18 11:23:51 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
    updated: 10.10.2.6:44949 is added to
    blk_1077143665_3402990{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-ab36f47b-c40a-4561-b4ad-8f7258257691:NORMAL:
    10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-48499fec-79cd-48df-943a-4e5011dc9f9a:NORMAL:
    10.10.2.1:44949
    RBW],
    ReplicaUC[[DISK]DS-3f5fb044-538f-4977-a24d-349c8496e77d:NORMAL:
    10.10.2.3:44949
    RBW]]}
    size 0
    16/06/18 11:23:51 INFO hdfs.StateChange: BLOCK* allocate
    blk_1077143674_3402999{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
    primaryNodeIndex=-1,

    replicas=[ReplicaUC[[DISK]DS-8af58951-0eb7-4244-b76a-9356ad832a93:NORMAL:
    10.10.2.6:44949
    RBW],
    ReplicaUC[[DISK]DS-6095b306-487e-48d7-948e-84ac5ef16aa4:NORMAL:
    10.10.2.9:44949
    RBW],
    ReplicaUC[[DISK]DS-9b2f64f6-bbac-4ed5-a1f9-9366129d196b:NORMAL:
    10.10.2.1:44949
    RBW]]}
    for

    /hbase/tweetTable-2016-02/d654d01588e8a46d7050852978f8eaf9/.tmp/14bc58b73bd74cf28cf7e43b2304e4d6
    Appreciate any insight on this.

    Regards,
    Chathuri

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedJun 18, '16 at 3:30p
activeJun 18, '16 at 5:14p
posts3
users2
websitehbase.apache.org

2 users in discussion

Chathuri Wimalasena: 2 posts Ted Yu: 1 post

People

Translate

site design / logo © 2018 Grokbase