Grokbase Groups HBase user May 2011

[HBase-user] mslab enabled jvm crash

Jack Levin
May 26, 2011 at 7:44 am
Wayne, I think you are hitting fragmentation, how often do you flush?
Can you share memstore flush graphs?
Here is ours:

We run at 12G Heap, 20% memstore size, 50% blockcache, have recently
added incremental mode to combat too frequent ParNew GCs:

export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log \
-XX:+CMSIncrementalMode \
-XX:+CMSIncrementalPacing \

On Wed, May 25, 2011 at 5:55 PM, Wayne wrote:
We are using std thrift from python. All writes are batched into usually 30k
writes per batch. The writes are small double/varchar(100) type values. Our
current write performance is fine for our needs...our concern is that they
are not sustainable over time given the GC timeouts.

Per the 4 items above, yes the staff is biased (obviously), the OS is what
2/3+ linux servers are running, the machines are Super Micro twins with 6
sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly non-standard).
We are plain vanilla. Our workload may be non-standard in that it is NOT
map-reduce. It is an evenly distributed q based home spun parallel
processing framework.

We are not a Java shop, and do not want to become one. I think to push the
limits and do well with hadoop+hdfs you have to buy into Java and have deep
skills there. We are database experts looking only for an open source
scalable database. We are a python shop and have no interest in digging deep
into java and the jvm.

On Wed, May 25, 2011 at 8:07 PM, Ted Dunning wrote:

How large are these writes?

Are you using asynchbase or other alternative client implementation?

Are you batching updates?
On Wed, May 25, 2011 at 2:44 PM, Wayne wrote:

What are your write levels? We are pushing 30-40k writes/sec/node on 1
nodes for 24-36-48-72 hours straight. We have only 4 writers per node so we
are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load is
max 50% including some app code, and memory is the 8g JVM out of 24G. We
our production as 20TB in MySQL and see 90% disk utilization for hours
day. A database must be able to accept being pounded 24/7. If the hardware
can handle it so should the database...this is not true for Java based
databases. The reality is that Java is not nearly ready for what a real
database will expect of it. Sure we could "back off" the volume and add 100
more nodes like Cassandra requires but then we might as well have used
something else given that hardware spend.

Our problem is that we have invested so much time with Hbase that it is
for us to walk away and go to the sharded PostgreSQL we should have used 9
months back. Sorry for the negativity but considering giving up after
invested all of this time is painful.

On Wed, May 25, 2011 at 4:21 PM, Erik Onnen wrote:

On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <>
It should be recognized that your experiences are a bit out of the
here.  Many hbase installations use more recent JVM's without
Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
common to never see a full GC across an 8GB heap.

What we never see is a ParNew taking .1 seconds, they're usually .01
and we never have full collections lasting 92 seconds. The only time
I've ever seen a JVM at 8GB take that long is when running on puny
(read virtualized) cores or when there are Ubuntu kernel bugs at play.
The same is true for our Cassandra deploys as well.

Search Discussions

Discussion Posts


Follow ups

Related Discussions