Grokbase Groups HBase dev March 2011
FAQ
Hi,
See email thread 'One of the regionserver aborted, then the master shut down
itself' for background.
I am evaluating various ways of trimming the memory footprint of RegionLoad
because there would be so many regions in production cluster.

Looking at field memstoreSizeMB of RegionLoad, I only found this reference -
AvroUtil.hslToASL()
Load balancer currently isn't checking this metric. And HRegion has
memstoreSize field.

I wonder whether we can trim field memstoreSizeMB off RegionLoad.

Please comment.

Search Discussions

  • Ryan Rawson at Mar 17, 2011 at 9:37 pm
    How much memory does profiling indicating these objects use? How much
    are you expecting to save?

    Saving 4-8 bytes even on a 10k region cluster is still only 80k of
    ram, not really significant.

    On Thu, Mar 17, 2011 at 2:32 PM, Ted Yu wrote:
    Hi,
    See email thread 'One of the regionserver aborted, then the master shut down
    itself' for background.
    I am evaluating various ways of trimming the memory footprint of RegionLoad
    because there would be so many regions in production cluster.

    Looking at field memstoreSizeMB of RegionLoad, I only found this reference -
    AvroUtil.hslToASL()
    Load balancer currently isn't checking this metric. And HRegion has
    memstoreSize field.

    I wonder whether we can trim field memstoreSizeMB off RegionLoad.

    Please comment.
  • Andrew Purtell at Mar 17, 2011 at 9:47 pm
    memstoreSizeMB is part of the output printed by the shell when you do status 'detailed'.

    I use that.

    Isn't that information useful to others?

    - Andy

    --- On Thu, 3/17/11, Ryan Rawson wrote:
    From: Ryan Rawson <ryanobjc@gmail.com>
    Subject: Re: trimming RegionLoad fields
    To: dev@hbase.apache.org
    Cc: "Ted Yu" <yuzhihong@gmail.com>
    Date: Thursday, March 17, 2011, 2:37 PM
    How much memory does profiling
    indicating these objects use?  How much
    are you expecting to save?

    Saving 4-8 bytes even on a 10k region cluster is still only
    80k of ram, not really significant.

    On Thu, Mar 17, 2011 at 2:32 PM, Ted Yu wrote:
    Hi,
    See email thread 'One of the regionserver aborted,
    then the master shut down
    itself' for background.
    I am evaluating various ways of trimming the memory
    footprint of RegionLoad
    because there would be so many regions in production cluster.
    Looking at field memstoreSizeMB of RegionLoad, I only
    found this reference -
    AvroUtil.hslToASL()
    Load balancer currently isn't checking this metric.
    And HRegion has
    memstoreSize field.

    I wonder whether we can trim field memstoreSizeMB off
    RegionLoad.
    Please comment.
  • Ryan Rawson at Mar 17, 2011 at 9:53 pm
    Without solid evidence of "we'll be saving X megabytes" I don't see a
    compelling reason to hacking that stuff out yet.

    We sort of need a better out-of-the-box monitoring system. One idea I
    had was to embed OpenTSDB inside the HMaster. This way OpenTSDB would
    store info about a HBase cluster back in the same cluster it monitors.
    While this may sound "weird" I think it makes sense because every
    great database system provides strong self monitoring tools. Eg:
    Oracle, etc. Due to the LGPL, this is not currently viable. Perhaps
    there is an alternative floating out there we can ship with? And not
    ganglia :-)
    On Thu, Mar 17, 2011 at 2:47 PM, Andrew Purtell wrote:
    memstoreSizeMB is part of the output printed by the shell when you do status 'detailed'.

    I use that.

    Isn't that information useful to others?

    - Andy

    --- On Thu, 3/17/11, Ryan Rawson wrote:
    From: Ryan Rawson <ryanobjc@gmail.com>
    Subject: Re: trimming RegionLoad fields
    To: dev@hbase.apache.org
    Cc: "Ted Yu" <yuzhihong@gmail.com>
    Date: Thursday, March 17, 2011, 2:37 PM
    How much memory does profiling
    indicating these objects use?  How much
    are you expecting to save?

    Saving 4-8 bytes even on a 10k region cluster is still only
    80k of ram, not really significant.


    On Thu, Mar 17, 2011 at 2:32 PM, Ted Yu <yuzhihong@gmail.com>
    wrote:
    Hi,
    See email thread 'One of the regionserver aborted,
    then the master shut down
    itself' for background.
    I am evaluating various ways of trimming the memory
    footprint of RegionLoad
    because there would be so many regions in production cluster.
    Looking at field memstoreSizeMB of RegionLoad, I only
    found this reference -
    AvroUtil.hslToASL()
    Load balancer currently isn't checking this metric.
    And HRegion has
    memstoreSize field.

    I wonder whether we can trim field memstoreSizeMB off
    RegionLoad.
    Please comment.

  • Ted Yu at Mar 17, 2011 at 9:53 pm
    I agree that it is useful information.
    Can we retrieve it from HRegion which is in HRegionServer.onlineRegions ?

    protected final Map<String, HRegion> onlineRegions =
    new HashMap<String, HRegion>();

    Thanks
    On Thu, Mar 17, 2011 at 2:47 PM, Andrew Purtell wrote:

    memstoreSizeMB is part of the output printed by the shell when you do
    status 'detailed'.

    I use that.

    Isn't that information useful to others?

    - Andy

    --- On Thu, 3/17/11, Ryan Rawson wrote:
    From: Ryan Rawson <ryanobjc@gmail.com>
    Subject: Re: trimming RegionLoad fields
    To: dev@hbase.apache.org
    Cc: "Ted Yu" <yuzhihong@gmail.com>
    Date: Thursday, March 17, 2011, 2:37 PM
    How much memory does profiling
    indicating these objects use? How much
    are you expecting to save?

    Saving 4-8 bytes even on a 10k region cluster is still only
    80k of ram, not really significant.


    On Thu, Mar 17, 2011 at 2:32 PM, Ted Yu <yuzhihong@gmail.com>
    wrote:
    Hi,
    See email thread 'One of the regionserver aborted,
    then the master shut down
    itself' for background.
    I am evaluating various ways of trimming the memory
    footprint of RegionLoad
    because there would be so many regions in production cluster.
    Looking at field memstoreSizeMB of RegionLoad, I only
    found this reference -
    AvroUtil.hslToASL()
    Load balancer currently isn't checking this metric.
    And HRegion has
    memstoreSize field.

    I wonder whether we can trim field memstoreSizeMB off
    RegionLoad.
    Please comment.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedMar 17, '11 at 9:32p
activeMar 17, '11 at 9:53p
posts5
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase