FAQ
Hello all,

I'm trying to get a new HBase cluster up and running on top of an
existing 5 node Hadoop cluster.

I've basically set it up such that the HBase master is running on the
same machine as the name node, and the region servers are running on
the other 4 machines as slaves.

I'm trying to populate a single table with about 10GB of data via a
map/reduce job, and I'm noticing that the region servers are running
out of memory.

Any pointers on what could be going on here?

Thanks,
Ryan

Search Discussions

  • Stack at Dec 19, 2008 at 7:09 pm
    Your cells are small? If so, please up your heap size (See
    $HBASE_HOME/conf/hbase-env.sh. See HADOOP_HEAPSIZE). This should be
    better in 0.19.0 (See HBASE-900).
    Thanks,
    St.Ack

    Ryan LeCompte wrote:
    Hello all,

    I'm trying to get a new HBase cluster up and running on top of an
    existing 5 node Hadoop cluster.

    I've basically set it up such that the HBase master is running on the
    same machine as the name node, and the region servers are running on
    the other 4 machines as slaves.

    I'm trying to populate a single table with about 10GB of data via a
    map/reduce job, and I'm noticing that the region servers are running
    out of memory.

    Any pointers on what could be going on here?

    Thanks,
    Ryan
  • Ryan LeCompte at Dec 19, 2008 at 7:11 pm
    Thanks, I'll give it a shot.

    On Fri, Dec 19, 2008 at 2:09 PM, stack wrote:
    Your cells are small? If so, please up your heap size (See
    $HBASE_HOME/conf/hbase-env.sh. See HADOOP_HEAPSIZE). This should be better
    in 0.19.0 (See HBASE-900).
    Thanks,
    St.Ack

    Ryan LeCompte wrote:
    Hello all,

    I'm trying to get a new HBase cluster up and running on top of an
    existing 5 node Hadoop cluster.

    I've basically set it up such that the HBase master is running on the
    same machine as the name node, and the region servers are running on
    the other 4 machines as slaves.

    I'm trying to populate a single table with about 10GB of data via a
    map/reduce job, and I'm noticing that the region servers are running
    out of memory.

    Any pointers on what could be going on here?

    Thanks,
    Ryan
  • Ryan LeCompte at Dec 19, 2008 at 9:06 pm
    So, I tried giving each of the region servers 2GB and also tried to
    limit the number of cells/columns I'm creating, but the memory still
    maxes out at 2GB eventually runs into OOME.

    Basically, I'm just trying to store about 5 or 10 "cells" under a
    column, where the values are very small (say 1 or 5 or 10 characters),
    and also the cells may not always exist, so it would be a sparse
    matrix.

    I'm wondering if I should just be storing serializable custom Java
    objects as cell values instead where the object contains the
    attributes that I'm currently trying to store as individual
    columns/cell values. Some of the attributes would be null if they're
    not present. I am not sure if there is any benefit to that.

    Would this help at all with the memory issues? Or should I downgrade
    from using HBase 0.18.1/ try different release?

    Thanks,
    Ryan
    On Fri, Dec 19, 2008 at 2:10 PM, Ryan LeCompte wrote:
    Thanks, I'll give it a shot.

    On Fri, Dec 19, 2008 at 2:09 PM, stack wrote:
    Your cells are small? If so, please up your heap size (See
    $HBASE_HOME/conf/hbase-env.sh. See HADOOP_HEAPSIZE). This should be better
    in 0.19.0 (See HBASE-900).
    Thanks,
    St.Ack

    Ryan LeCompte wrote:
    Hello all,

    I'm trying to get a new HBase cluster up and running on top of an
    existing 5 node Hadoop cluster.

    I've basically set it up such that the HBase master is running on the
    same machine as the name node, and the region servers are running on
    the other 4 machines as slaves.

    I'm trying to populate a single table with about 10GB of data via a
    map/reduce job, and I'm noticing that the region servers are running
    out of memory.

    Any pointers on what could be going on here?

    Thanks,
    Ryan
  • Stack at Dec 19, 2008 at 9:15 pm
    Small cell sizes use up loads of memory. See HBASE-900 for more on this.

    Other things to try:

    + hbase.io.index.interval is 32 by default. Set it to 1024 or larger
    in your case. Access may be a little slower -- but maybe not so bad
    since your cells are so small -- but the memory-resident indices will be
    smaller.
    + Set hbase.regionserver.globalMemcacheLimit down from the hard-coded
    512MB to something like 384MB. This will allow more of the RAM serving
    indices.
    + If you can, update to 0.19.0 hadoop and hbase TRUNK. In 0.18.0, our
    accounting of RAM-resident items is off.

    Let us know how it goes.
    St.Ack


    Ryan LeCompte wrote:
    So, I tried giving each of the region servers 2GB and also tried to
    limit the number of cells/columns I'm creating, but the memory still
    maxes out at 2GB eventually runs into OOME.

    Basically, I'm just trying to store about 5 or 10 "cells" under a
    column, where the values are very small (say 1 or 5 or 10 characters),
    and also the cells may not always exist, so it would be a sparse
    matrix.

    I'm wondering if I should just be storing serializable custom Java
    objects as cell values instead where the object contains the
    attributes that I'm currently trying to store as individual
    columns/cell values. Some of the attributes would be null if they're
    not present. I am not sure if there is any benefit to that.

    Would this help at all with the memory issues? Or should I downgrade
    from using HBase 0.18.1/ try different release?

    Thanks,
    Ryan
    On Fri, Dec 19, 2008 at 2:10 PM, Ryan LeCompte wrote:

    Thanks, I'll give it a shot.

    On Fri, Dec 19, 2008 at 2:09 PM, stack wrote:

    Your cells are small? If so, please up your heap size (See
    $HBASE_HOME/conf/hbase-env.sh. See HADOOP_HEAPSIZE). This should be better
    in 0.19.0 (See HBASE-900).
    Thanks,
    St.Ack

    Ryan LeCompte wrote:
    Hello all,

    I'm trying to get a new HBase cluster up and running on top of an
    existing 5 node Hadoop cluster.

    I've basically set it up such that the HBase master is running on the
    same machine as the name node, and the region servers are running on
    the other 4 machines as slaves.

    I'm trying to populate a single table with about 10GB of data via a
    map/reduce job, and I'm noticing that the region servers are running
    out of memory.

    Any pointers on what could be going on here?

    Thanks,
    Ryan
  • Stack at Dec 19, 2008 at 9:19 pm

    stack wrote:
    Small cell sizes use up loads of memory. See HBASE-900 for more on this.

    Other things to try:

    + hbase.io.index.interval is 32 by default. Set it to 1024 or larger
    in your case. Access may be a little slower -- but maybe not so bad
    since your cells are so small -- but the memory-resident indices will
    be smaller.
    Oh, pardon me, this is broke in 0.18.x hbase. See HBASE-981. Changing
    the value has no effect.
    St.Ack
  • Ryan LeCompte at Dec 19, 2008 at 9:57 pm
    Stack,

    Thanks for responding. I'm going to experiment with using a simple
    Java object as the cell value as opposed to little tiny string cell
    values and see if that helps. I can't afford to keep increasing the
    memory since I'll eventually run out of space for my other map/reduce
    jobs.

    Will keep you all posted.

    Thanks,
    Ryan

    On Fri, Dec 19, 2008 at 4:18 PM, stack wrote:
    stack wrote:
    Small cell sizes use up loads of memory. See HBASE-900 for more on this.

    Other things to try:

    + hbase.io.index.interval is 32 by default. Set it to 1024 or larger in
    your case. Access may be a little slower -- but maybe not so bad since your
    cells are so small -- but the memory-resident indices will be smaller.
    Oh, pardon me, this is broke in 0.18.x hbase. See HBASE-981. Changing the
    value has no effect.
    St.Ack
  • Ryan LeCompte at Dec 19, 2008 at 11:53 pm
    Hey Stack,

    Okay, I was able to get Hadoop 0.19 up and running with Hbase-trunk.
    It seems to startup fine, however now when I connect to the hbase
    shell and do a simple "list" or try to create a table, I get the
    following almost immediately in the hbase master log files:

    2008-12-19 18:49:47,408 WARN org.apache.hadoop.ipc.HBaseServer: Out of
    Memory in server select
    java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:142)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:846)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:813)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:399)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
    2008-12-19 18:49:49,888 INFO
    org.apache.hadoop.hbase.master.BaseScanner: All 0 .META. region(s)
    scanned


    Any ideas?

    Thanks,
    Ryan

    On Fri, Dec 19, 2008 at 4:56 PM, Ryan LeCompte wrote:
    Stack,

    Thanks for responding. I'm going to experiment with using a simple
    Java object as the cell value as opposed to little tiny string cell
    values and see if that helps. I can't afford to keep increasing the
    memory since I'll eventually run out of space for my other map/reduce
    jobs.

    Will keep you all posted.

    Thanks,
    Ryan

    On Fri, Dec 19, 2008 at 4:18 PM, stack wrote:
    stack wrote:
    Small cell sizes use up loads of memory. See HBASE-900 for more on this.

    Other things to try:

    + hbase.io.index.interval is 32 by default. Set it to 1024 or larger in
    your case. Access may be a little slower -- but maybe not so bad since your
    cells are so small -- but the memory-resident indices will be smaller.
    Oh, pardon me, this is broke in 0.18.x hbase. See HBASE-981. Changing the
    value has no effect.
    St.Ack
  • Andrew Purtell at Dec 20, 2008 at 12:59 am
    That is almost certainly a HBase jar version mismatch
    between the master and the client.

    I had this problem once when the jars for my master and one
    regionserver were out of sync.

    - Andy
    From: Ryan LeCompte <lecompte@gmail.com>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 3:53 PM
    Hey Stack,

    Okay, I was able to get Hadoop 0.19 up and running with
    Hbase-trunk. It seems to startup fine, however now when
    I connect to the hbase shell and do a simple "list" or
    try to create a table, I get the
    following almost immediately in the hbase master log files:

    2008-12-19 18:49:47,408 WARN
    org.apache.hadoop.ipc.HBaseServer: Out of
    Memory in server select
    java.lang.OutOfMemoryError: Java heap space
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:142)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:846)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:813)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:399)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
    2008-12-19 18:49:49,888 INFO
    org.apache.hadoop.hbase.master.BaseScanner: All 0 .META.
    region(s)
    scanned


    Any ideas?

    Thanks,
    Ryan
  • Ryan LeCompte at Dec 20, 2008 at 1:31 am
    Really? Hmm.... I just tested all 5 nodes and the JARs are the same,
    since I've just rsync'd them over. The server doesn't spit out any
    errors when it starts up...

    Could this be an hadoop/hbase jar mixmatch?

    On Fri, Dec 19, 2008 at 7:58 PM, Andrew Purtell wrote:
    That is almost certainly a HBase jar version mismatch
    between the master and the client.

    I had this problem once when the jars for my master and one
    regionserver were out of sync.

    - Andy
    From: Ryan LeCompte <lecompte@gmail.com>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 3:53 PM
    Hey Stack,

    Okay, I was able to get Hadoop 0.19 up and running with
    Hbase-trunk. It seems to startup fine, however now when
    I connect to the hbase shell and do a simple "list" or
    try to create a table, I get the
    following almost immediately in the hbase master log files:

    2008-12-19 18:49:47,408 WARN
    org.apache.hadoop.ipc.HBaseServer: Out of
    Memory in server select
    java.lang.OutOfMemoryError: Java heap space
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:142)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:846)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:813)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:399)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
    2008-12-19 18:49:49,888 INFO
    org.apache.hadoop.hbase.master.BaseScanner: All 0 .META.
    region(s)
    scanned


    Any ideas?

    Thanks,
    Ryan


  • Ryan LeCompte at Dec 20, 2008 at 1:36 am
    Ah, you're right! Jar mismatch. Thanks :-)

    On Fri, Dec 19, 2008 at 8:31 PM, Ryan LeCompte wrote:
    Really? Hmm.... I just tested all 5 nodes and the JARs are the same,
    since I've just rsync'd them over. The server doesn't spit out any
    errors when it starts up...

    Could this be an hadoop/hbase jar mixmatch?

    On Fri, Dec 19, 2008 at 7:58 PM, Andrew Purtell wrote:
    That is almost certainly a HBase jar version mismatch
    between the master and the client.

    I had this problem once when the jars for my master and one
    regionserver were out of sync.

    - Andy
    From: Ryan LeCompte <lecompte@gmail.com>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 3:53 PM
    Hey Stack,

    Okay, I was able to get Hadoop 0.19 up and running with
    Hbase-trunk. It seems to startup fine, however now when
    I connect to the hbase shell and do a simple "list" or
    try to create a table, I get the
    following almost immediately in the hbase master log files:

    2008-12-19 18:49:47,408 WARN
    org.apache.hadoop.ipc.HBaseServer: Out of
    Memory in server select
    java.lang.OutOfMemoryError: Java heap space
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Invocation.readFields(HBaseRPC.java:142)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:846)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:813)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:399)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
    2008-12-19 18:49:49,888 INFO
    org.apache.hadoop.hbase.master.BaseScanner: All 0 .META.
    region(s)
    scanned


    Any ideas?

    Thanks,
    Ryan


  • Andrew Purtell at Dec 20, 2008 at 12:42 am
    Does it make sense to backport HBASE-900 to 0.18.2?

    - Andy
    From: Ryan LeCompte <lecompte@gmail.com>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 1:06 PM
    So, I tried giving each of the region servers 2GB and also
    tried to limit the number of cells/columns I'm creating,
    but the memory still maxes out at 2GB eventually runs into
    OOME.
  • Stack at Dec 20, 2008 at 2:22 am
    I was just thinking on this. Testing, changing the index interval has
    the biggest direct effect on memory used. Currently its 32 which seems
    to be fine for cells of 1K and greater. With 32, you can load a decent
    amount of regions per regionserver. Smaller cells seem common enough
    though. Here interval should be the hadoop, as opposed to hbase,
    default of 128 or even 1024 (Its 'low' to help improve access times).
    It seems that for the case where cells are < 10 bytes or so, interval
    should be 1k or even 4k. Should we just be conservative in defaults
    since OOME is worse than slow access times.

    Backporting HBASE-900 might be a bit much for 0.18.2. We should for
    sure backport the fix that makes it so you can set the index interval to
    0.18 branch.

    St.Ack

    Andrew Purtell wrote:
    Does it make sense to backport HBASE-900 to 0.18.2?

    - Andy

    From: Ryan LeCompte <lecompte@gmail.com>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 1:06 PM
    So, I tried giving each of the region servers 2GB and also
    tried to limit the number of cells/columns I'm creating,
    but the memory still maxes out at 2GB eventually runs into
    OOME.


  • Andrew Purtell at Dec 20, 2008 at 3:13 am
    Based on this, leaving the Hadoop default (128) might be the
    way to go.

    Later, maybe it would make sense to dynamically set the index
    interval based on the distribution of cell sizes in the
    mapfile at some future time, according to some parameterized
    formula that could be adjusted with config variable(s). This
    could be done during compaction. Would make sense to also
    consider the distribution of key lengths. Or there could be
    other similar tricks implemented to keep index sizes down.

    In my opinion, for 0.20.0, MapFile should be brought local so
    we can begin hacking it over time into a new file format. I
    was thinking that designing and/or implementing a wholly new
    format such as TFile would block I/O improvements for a long
    time.

    - Andy
    From: stack <stack@duboce.net>
    Subject: Re: Region server memory requirements
    To: hbase-user@hadoop.apache.org
    Date: Friday, December 19, 2008, 6:22 PM
    I was just thinking on this. Testing, changing the index
    interval has the biggest direct effect on memory used.
    Currently its 32 which seems to be fine for cells of 1K and
    greater. With 32, you can load a decent amount of regions
    per regionserver. Smaller cells seem common enough though.
    Here interval should be the hadoop, as opposed to hbase,
    default of 128 or even 1024 (Its 'low' to help
    improve access times). It seems that for the case where
    cells are < 10 bytes or so, interval should be 1k or even
    4k. Should we just be conservative in defaults since OOME
    is worse than slow access times.

    Backporting HBASE-900 might be a bit much for 0.18.2. We
    should for sure backport the fix that makes it so you can
    set the index interval to 0.18 branch.

    St.Ack

    Andrew Purtell wrote:
    Does it make sense to backport HBASE-900 to 0.18.2?

    - Andy
  • Stack at Dec 20, 2008 at 4:02 am

    Andrew Purtell wrote:
    Based on this, leaving the Hadoop default (128) might be the
    way to go.
    Sounds good. I made HBASE-1070 to do above for TRUNK and branch.
    Later, maybe it would make sense to dynamically set the index
    interval based on the distribution of cell sizes in the
    mapfile at some future time, according to some parameterized
    formula that could be adjusted with config variable(s). This
    could be done during compaction. Would make sense to also
    consider the distribution of key lengths. Or there could be
    other similar tricks implemented to keep index sizes down.
    Made HBASE-1071. We should be able to do it at flush time, not just
    compacting, since we have count of keys and could keep running tally on
    memcache insert of notable attributes of key so we had these to plugin
    to the formula at flush time.
    In my opinion, for 0.20.0, MapFile should be brought local so
    we can begin hacking it over time into a new file format. I
    was thinking that designing and/or implementing a wholly new
    format such as TFile would block I/O improvements for a long
    time.
    I don't know. This would be the safer tack for sure, but lets at least
    keep open the possibility of our moving to a completely new file format
    in 0.20.0 timeframe.

    St.Ack
  • Billy Pearson at Dec 22, 2008 at 7:41 pm
    hey guys there is a var in hadoop that can help with out having to change
    the index int its io.map.index.skip
    this can be changed to lower memory usage without having to wait until the
    map files are compacted again and you can change as needed.

    "stack" <stack@duboce.net> wrote in message
    news:494C6E27.7030105@duboce.net...
    Andrew Purtell wrote:
    Based on this, leaving the Hadoop default (128) might be the
    way to go.
    Sounds good. I made HBASE-1070 to do above for TRUNK and branch.
    Later, maybe it would make sense to dynamically set the index
    interval based on the distribution of cell sizes in the mapfile at some
    future time, according to some parameterized
    formula that could be adjusted with config variable(s). This
    could be done during compaction. Would make sense to also
    consider the distribution of key lengths. Or there could be
    other similar tricks implemented to keep index sizes down.
    Made HBASE-1071. We should be able to do it at flush time, not just
    compacting, since we have count of keys and could keep running tally on
    memcache insert of notable attributes of key so we had these to plugin to
    the formula at flush time.
    In my opinion, for 0.20.0, MapFile should be brought local so
    we can begin hacking it over time into a new file format. I
    was thinking that designing and/or implementing a wholly new
    format such as TFile would block I/O improvements for a long
    time.
    I don't know. This would be the safer tack for sure, but lets at least
    keep open the possibility of our moving to a completely new file format in
    0.20.0 timeframe.

    St.Ack
  • Stack at Dec 22, 2008 at 7:59 pm

    Billy Pearson wrote:
    hey guys there is a var in hadoop that can help with out having to
    change the index int its io.map.index.skip
    this can be changed to lower memory usage without having to wait until
    the map files are compacted again and you can change as needed.
    Yeah.

    Here is what our hbase.io.index.interval property currently says:

    <property>
    <name>hbase.io.index.interval</name>
    <value>128</value>
    <description>The interval at which we record offsets in hbase
    store files/mapfiles. Default for stock mapfiles is 128. Index
    files are read into memory. If there are many of them, could prove
    a burden. If so play with the hadoop io.map.index.skip property and
    skip every nth index member when reading back the index into memory.
    Downside to high index interval is lowered access times.
    </description>
    </property>

    One approach might have been to keep writing at a small interval and
    then set index.skip to specify how much of the index to read in.

    St.Ack
    "stack" <stack@duboce.net> wrote in message
    news:494C6E27.7030105@duboce.net...
    Andrew Purtell wrote:
    Based on this, leaving the Hadoop default (128) might be the
    way to go.
    Sounds good. I made HBASE-1070 to do above for TRUNK and branch.
    Later, maybe it would make sense to dynamically set the index
    interval based on the distribution of cell sizes in the mapfile at
    some future time, according to some parameterized
    formula that could be adjusted with config variable(s). This
    could be done during compaction. Would make sense to also
    consider the distribution of key lengths. Or there could be
    other similar tricks implemented to keep index sizes down.
    Made HBASE-1071. We should be able to do it at flush time, not just
    compacting, since we have count of keys and could keep running tally
    on memcache insert of notable attributes of key so we had these to
    plugin to the formula at flush time.
    In my opinion, for 0.20.0, MapFile should be brought local so
    we can begin hacking it over time into a new file format. I
    was thinking that designing and/or implementing a wholly new
    format such as TFile would block I/O improvements for a long
    time.
    I don't know. This would be the safer tack for sure, but lets at
    least keep open the possibility of our moving to a completely new
    file format in 0.20.0 timeframe.

    St.Ack

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedDec 19, '08 at 7:06p
activeDec 22, '08 at 7:59p
posts17
users4
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase