[HBase-user] Region server memory requirements

Hello all,

I'm trying to get a new HBase cluster up and running on top of an
existing 5 node Hadoop cluster.

I've basically set it up such that the HBase master is running on the
same machine as the name node, and the region servers are running on
the other 4 machines as slaves.

I'm trying to populate a single table with about 10GB of data via a
map/reduce job, and I'm noticing that the region servers are running
out of memory.

Any pointers on what could be going on here?

Thanks,
Ryan

On Fri, Dec 19, 2008 at 2:09 PM, stack wrote:
Your cells are small? If so, please up your heap size (See
$HBASE_HOME/conf/hbase-env.sh. See HADOOP_HEAPSIZE). This should be better
in 0.19.0 (See HBASE-900).
Thanks,
St.Ack

Ryan LeCompte wrote:

On Fri, Dec 19, 2008 at 2:10 PM, Ryan LeCompte wrote:
Thanks, I'll give it a shot.

So, I tried giving each of the region servers 2GB and also tried to
limit the number of cells/columns I'm creating, but the memory still
maxes out at 2GB eventually runs into OOME.

Basically, I'm just trying to store about 5 or 10 "cells" under a
column, where the values are very small (say 1 or 5 or 10 characters),
and also the cells may not always exist, so it would be a sparse
matrix.

I'm wondering if I should just be storing serializable custom Java
objects as cell values instead where the object contains the
attributes that I'm currently trying to store as individual
columns/cell values. Some of the attributes would be null if they're
not present. I am not sure if there is any benefit to that.

Would this help at all with the memory issues? Or should I downgrade
from using HBase 0.18.1/ try different release?

Thanks,
Ryan

stack wrote:
Small cell sizes use up loads of memory. See HBASE-900 for more on this.

Other things to try:

+ hbase.io.index.interval is 32 by default. Set it to 1024 or larger
in your case. Access may be a little slower -- but maybe not so bad
since your cells are so small -- but the memory-resident indices will
be smaller.

On Fri, Dec 19, 2008 at 4:18 PM, stack wrote:
stack wrote:
Oh, pardon me, this is broke in 0.18.x hbase. See HBASE-981. Changing the
value has no effect.
St.Ack

On Fri, Dec 19, 2008 at 4:56 PM, Ryan LeCompte wrote:
Stack,

Thanks for responding. I'm going to experiment with using a simple
Java object as the cell value as opposed to little tiny string cell
values and see if that helps. I can't afford to keep increasing the
memory since I'll eventually run out of space for my other map/reduce
jobs.

Will keep you all posted.

Thanks,
Ryan

From: Ryan LeCompte <lecompte@gmail.com>
Subject: Re: Region server memory requirements
To: hbase-user@hadoop.apache.org
Date: Friday, December 19, 2008, 3:53 PM
Hey Stack,

Okay, I was able to get Hadoop 0.19 up and running with
Hbase-trunk. It seems to startup fine, however now when
I connect to the hbase shell and do a simple "list" or
try to create a table, I get the
following almost immediately in the hbase master log files:

2008-12-19 18:49:47,408 WARN
org.apache.hadoop.ipc.HBaseServer: Out of
Memory in server select
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:142)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:846)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:813)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:399)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
2008-12-19 18:49:49,888 INFO
org.apache.hadoop.hbase.master.BaseScanner: All 0 .META.
region(s)
scanned


Any ideas?

Thanks,
Ryan

On Fri, Dec 19, 2008 at 7:58 PM, Andrew Purtell wrote:
That is almost certainly a HBase jar version mismatch
between the master and the client.

I had this problem once when the jars for my master and one
regionserver were out of sync.

- Andy

On Fri, Dec 19, 2008 at 8:31 PM, Ryan LeCompte wrote:
Really? Hmm.... I just tested all 5 nodes and the JARs are the same,
since I've just rsync'd them over. The server doesn't spit out any
errors when it starts up...

Could this be an hadoop/hbase jar mixmatch?

From: Ryan LeCompte <lecompte@gmail.com>
Subject: Re: Region server memory requirements
To: hbase-user@hadoop.apache.org
Date: Friday, December 19, 2008, 1:06 PM
So, I tried giving each of the region servers 2GB and also
tried to limit the number of cells/columns I'm creating,
but the memory still maxes out at 2GB eventually runs into
OOME.

Does it make sense to backport HBASE-900 to 0.18.2?

- Andy

From: stack <stack@duboce.net>
Subject: Re: Region server memory requirements
To: hbase-user@hadoop.apache.org
Date: Friday, December 19, 2008, 6:22 PM
I was just thinking on this. Testing, changing the index
interval has the biggest direct effect on memory used.
Currently its 32 which seems to be fine for cells of 1K and
greater. With 32, you can load a decent amount of regions
per regionserver. Smaller cells seem common enough though.
Here interval should be the hadoop, as opposed to hbase,
default of 128 or even 1024 (Its 'low' to help
improve access times). It seems that for the case where
cells are < 10 bytes or so, interval should be 1k or even
4k. Should we just be conservative in defaults since OOME
is worse than slow access times.

Backporting HBASE-900 might be a bit much for 0.18.2. We
should for sure backport the fix that makes it so you can
set the index interval to 0.18 branch.

St.Ack

Andrew Purtell wrote:

Andrew Purtell wrote:
Based on this, leaving the Hadoop default (128) might be the
way to go.
Sounds good. I made HBASE-1070 to do above for TRUNK and branch.
Later, maybe it would make sense to dynamically set the index
interval based on the distribution of cell sizes in the
mapfile at some future time, according to some parameterized
formula that could be adjusted with config variable(s). This
could be done during compaction. Would make sense to also
consider the distribution of key lengths. Or there could be
other similar tricks implemented to keep index sizes down.

Andrew Purtell wrote:
Made HBASE-1071. We should be able to do it at flush time, not just
compacting, since we have count of keys and could keep running tally on
memcache insert of notable attributes of key so we had these to plugin to
the formula at flush time.
I don't know. This would be the safer tack for sure, but lets at least
keep open the possibility of our moving to a completely new file format in
0.20.0 timeframe.

St.Ack

Billy Pearson wrote:
hey guys there is a var in hadoop that can help with out having to
change the index int its io.map.index.skip
this can be changed to lower memory usage without having to wait until
the map files are compacted again and you can change as needed.

"stack" <stack@duboce.net> wrote in message
news:494C6E27.7030105@duboce.net...