Grokbase Groups HBase dev March 2008
FAQ
Two runs of PerformanceEvaluation from HBase trunk failed with region off-line error
-------------------------------------------------------------------------------------

Key: HBASE-485
URL: https://issues.apache.org/jira/browse/HBASE-485
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Jim Kellerman
Priority: Critical


Two runs of the PerformanceEvaluation job on our test cluster running HBase trunk failed with region offline error.

The following is an excerpt from the failed task's log:
{code}
2008-03-01 19:30:42,184 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
2008-03-01 19:30:42,188 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:30:52,196 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
2008-03-01 19:30:52,203 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:31:02,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:31:02,236 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
2008-03-01 19:31:02,237 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
2008-03-01 19:31:02,240 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:31:12,261 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
2008-03-01 19:31:12,270 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.lang.RuntimeException: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1019)
at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:753)
at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:465)
at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:333)
at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:529)
at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:178)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
Caused by: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:447)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:352)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:314)
at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:109)
at org.apache.hadoop.hbase.client.HTable$ServerCallable.instantiateServer(HTable.java:992)
at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1006)
... 8 more
{code}


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Jim Kellerman (JIRA) at Mar 1, 2008 at 11:21 pm
    [ https://issues.apache.org/jira/browse/HBASE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574161#action_12574161 ]

    Jim Kellerman commented on HBASE-485:
    -------------------------------------

    The interesting thing to note is the empty HRegionInfo in META
    Two runs of PerformanceEvaluation from HBase trunk failed with region off-line error
    -------------------------------------------------------------------------------------

    Key: HBASE-485
    URL: https://issues.apache.org/jira/browse/HBASE-485
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.2.0
    Reporter: Jim Kellerman
    Priority: Critical

    Two runs of the PerformanceEvaluation job on our test cluster running HBase trunk failed with region offline error.
    The following is an excerpt from the failed task's log:
    {code}
    2008-03-01 19:30:42,184 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:42,188 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,196 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:52,203 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,236 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:31:02,237 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:31:02,240 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,261 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,270 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
    java.lang.RuntimeException: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1019)
    at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:753)
    at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:465)
    at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:333)
    at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:529)
    at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:178)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
    Caused by: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:447)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:352)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:314)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:109)
    at org.apache.hadoop.hbase.client.HTable$ServerCallable.instantiateServer(HTable.java:992)
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1006)
    ... 8 more
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • stack (JIRA) at Mar 2, 2008 at 4:49 am
    [ https://issues.apache.org/jira/browse/HBASE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574185#action_12574185 ]

    stack commented on HBASE-485:
    -----------------------------

    Interesting. Looks like HBASE-17 (also happened under PE). Also seen in HBASE-471. Can you figure it Jim?
    Two runs of PerformanceEvaluation from HBase trunk failed with region off-line error
    -------------------------------------------------------------------------------------

    Key: HBASE-485
    URL: https://issues.apache.org/jira/browse/HBASE-485
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.2.0
    Reporter: Jim Kellerman
    Priority: Critical

    Two runs of the PerformanceEvaluation job on our test cluster running HBase trunk failed with region offline error.
    The following is an excerpt from the failed task's log:
    {code}
    2008-03-01 19:30:42,184 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:42,188 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,196 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:52,203 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,236 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:31:02,237 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:31:02,240 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,261 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,270 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
    java.lang.RuntimeException: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1019)
    at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:753)
    at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:465)
    at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:333)
    at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:529)
    at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:178)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
    Caused by: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:447)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:352)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:314)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:109)
    at org.apache.hadoop.hbase.client.HTable$ServerCallable.instantiateServer(HTable.java:992)
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1006)
    ... 8 more
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Mar 2, 2008 at 11:45 am
    [ https://issues.apache.org/jira/browse/HBASE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574210#action_12574210 ]

    Jim Kellerman commented on HBASE-485:
    -------------------------------------

    No, I can't figure out why it should have started failing. I did
    update to latest trunk but the only issues that are even remotely
    related are HBASE-473, and 428, neither of which should impact the
    performance evaluation test (I wouldn't think). It must be one of
    the very latest commits, because PE was working on Friday (though
    I hadn't updated then) and now it reliably gets the region off-line
    error.

    ---
    Jim Kellerman, Senior Engineer; Powerset


    sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=> 12574185#action_12574185 ]
    est.testRow(PerformanceEvaluation.java:465)
    manceEvaluation.java:333)
    k.map(PerformanceEvaluation.java:178)
    .locateRegionInMeta(HConnectionManager.java:447)
    .locateRegion(HConnectionManager.java:352)
    .locateRegion(HConnectionManager.java:314)
    teServer(HTable.java:992)

    No virus found in this outgoing message.
    Checked by AVG Free Edition.
    Version: 7.5.516 / Virus Database: 269.21.2/1305 - Release Date: 2/29/2008 6:32 PM


    Two runs of PerformanceEvaluation from HBase trunk failed with region off-line error
    -------------------------------------------------------------------------------------

    Key: HBASE-485
    URL: https://issues.apache.org/jira/browse/HBASE-485
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.2.0
    Reporter: Jim Kellerman
    Priority: Critical

    Two runs of the PerformanceEvaluation job on our test cluster running HBase trunk failed with region offline error.
    The following is an excerpt from the failed task's log:
    {code}
    2008-03-01 19:30:42,184 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:42,188 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,196 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:52,203 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,236 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:31:02,237 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:31:02,240 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,261 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,270 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
    java.lang.RuntimeException: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1019)
    at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:753)
    at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:465)
    at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:333)
    at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:529)
    at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:178)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
    Caused by: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:447)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:352)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:314)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:109)
    at org.apache.hadoop.hbase.client.HTable$ServerCallable.instantiateServer(HTable.java:992)
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1006)
    ... 8 more
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Jim Kellerman (JIRA) at Mar 3, 2008 at 3:46 am
    [ https://issues.apache.org/jira/browse/HBASE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jim Kellerman resolved HBASE-485.
    ---------------------------------

    Resolution: Cannot Reproduce

    Although it failed reliably. I cannot reproduce it now.
    Two runs of PerformanceEvaluation from HBase trunk failed with region off-line error
    -------------------------------------------------------------------------------------

    Key: HBASE-485
    URL: https://issues.apache.org/jira/browse/HBASE-485
    Project: Hadoop HBase
    Issue Type: Bug
    Affects Versions: 0.2.0
    Reporter: Jim Kellerman
    Priority: Critical

    Two runs of the PerformanceEvaluation job on our test cluster running HBase trunk failed with region offline error.
    The following is an excerpt from the failed task's log:
    {code}
    2008-03-01 19:30:42,184 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:42,186 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:42,188 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,196 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:30:52,200 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:30:52,203 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,219 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:02,236 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: reloading table servers because: HRegionInfo was null or empty in .META.
    2008-03-01 19:31:02,237 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Removed .META.,,1 from cache because of TestTable,0066060288,999999999999999
    2008-03-01 19:31:02,240 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT regionname: -ROOT-,,0, startKey: <>, endKey: <>, encodedName: 70236052, tableDesc: {name: -ROOT-, families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,261 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found possible location for TestTable,0066060288,999999999999999, address: 208.76.44.140:8020, regioninfo: regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc: {name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, bloom filter: none}}}
    2008-03-01 19:31:12,270 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
    java.lang.RuntimeException: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1019)
    at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:753)
    at org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:465)
    at org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:333)
    at org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:529)
    at org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:178)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
    Caused by: java.lang.IllegalStateException: region offline: TestTable,0012639235,1204398750147
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:447)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:352)
    at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:314)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:109)
    at org.apache.hadoop.hbase.client.HTable$ServerCallable.instantiateServer(HTable.java:992)
    at org.apache.hadoop.hbase.client.HTable.getRegionServerWithRetries(HTable.java:1006)
    ... 8 more
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedMar 1, '08 at 11:19p
activeMar 3, '08 at 3:46a
posts5
users1
websitehbase.apache.org

1 user in discussion

Jim Kellerman (JIRA): 5 posts

People

Translate

site design / logo © 2022 Grokbase