Grokbase Groups HBase dev July 2009
FAQ
Hi,

I am using the latest HBase trunk on top of hadoop 0.20.0. I have a MR
job that digests records and stores them in a table in HBase.

Hadoop/HBase/ZooKeeper are deployed on a cluster of 5 machines (Linux,
64bits, 16Gb or RAM, 2x1Tb of disk).

After some time, the MR job fails exceptions similar to:

"org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server Some server for region ...."

This looks like the problem described in HBASE-1603 except the split
of the region that the job fails to access occurs very close to the
time of the failure.

HBASE-1615 is applied to the version of HBase trunk this problem occurs on.

Anyone else experiencing that?

Mathias.

Search Discussions

  • Stack at Jul 22, 2009 at 10:49 pm
    You've never run with an older version of TRUNK? Only a recent one, one that
    had 1615 in it?

    I might have seen this in a recent test run. Let me retry. At least there
    is better debug since hbase-1603.

    Thanks for writing the list.

    St.Ack
    On Wed, Jul 22, 2009 at 9:34 AM, Mathias Herberts wrote:

    Hi,

    I am using the latest HBase trunk on top of hadoop 0.20.0. I have a MR
    job that digests records and stores them in a table in HBase.

    Hadoop/HBase/ZooKeeper are deployed on a cluster of 5 machines (Linux,
    64bits, 16Gb or RAM, 2x1Tb of disk).

    After some time, the MR job fails exceptions similar to:

    "org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
    contact region server Some server for region ...."

    This looks like the problem described in HBASE-1603 except the split
    of the region that the job fails to access occurs very close to the
    time of the failure.

    HBASE-1615 is applied to the version of HBase trunk this problem occurs on.

    Anyone else experiencing that?

    Mathias.
  • Mathias Herberts at Jul 23, 2009 at 6:41 am

    You've never run with an older version of TRUNK? Only a recent one, one that
    had 1615 in it? Nope.
    I might have seen this in a recent test run.  Let me retry.  At least there
    is better debug since hbase-1603.
    I think the problem is indeed related to a split as it occurs less
    and less as I rerun the MR job (and thus the region count is already
    high and less splits are needed).

    If I store records in HBase from the mappers, the problem is even
    worse when starting with an empty table as records are not sorted and
    more splits occur more rapidly.

    Mathias.
  • Ryan Rawson at Jul 23, 2009 at 6:45 am
    how many cpus and cores does your system have?

    you cant run a map-reduce mapper, dfs, hbase on 2 cpus as we discovered recently

    On Wed, Jul 22, 2009 at 11:42 PM, Mathias
    Herbertswrote:
    You've never run with an older version of TRUNK? Only a recent one, one that
    had 1615 in it? Nope.
    I might have seen this in a recent test run.  Let me retry.  At least there
    is better debug since hbase-1603.
    I think the problem is indeed related to a split as it  occurs less
    and less as I rerun the MR job (and thus the region count is already
    high and less splits are needed).

    If I store records in HBase from the mappers, the problem is even
    worse when starting with an empty table as records are not sorted and
    more splits occur more rapidly.

    Mathias.
  • Mathias Herberts at Jul 23, 2009 at 6:48 am

    On Thu, Jul 23, 2009 at 08:46, Ryan Rawsonwrote:
    how many cpus and cores does your system have?

    you cant run a map-reduce mapper, dfs, hbase on 2 cpus as we discovered recently
    Each machine is a bi Quad Core, so the total number of CPUs in the
    cluster is 5x2 = 10 and the total number of cores 5x2x4 = 40.

    Is the limit you mention per machine or per cluster?

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedJul 22, '09 at 4:34p
activeJul 23, '09 at 6:48a
posts5
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase