Grokbase Groups HBase dev July 2012
FAQ
Hi Devs,

How about introducing multiple GC tuning profiles (like STANDALONE, SMALL,
MEDIUM, LARGE ...) in HBase itself?
Profiles can be categorized based on cluster size, usage, resources
available (Memory, CPU).

Why tuning profiles?
-> Its tedious & error-prone to find, edit some parameter in script file.
-> While editing manually and I'm sure we miss one or the other parameters
in some cluster machine.
-> These profiles boosts the user's confidence rather tune himself by trial
and error for each parameter.


I know I will get the following immediate answers. :-)
-> it's the responsibility of user to tune GC
-> it depends on usage
-> it depends on environment (machine resources available)
-> we do provide recommendations in hbase book
-> there are so many resources over web about tuning. Why don't you refer to
that?
-> so on...

I humbly accept all the above reasons but still I wanted to put forward the
idea of tuning profiles and get your opinion.
--
Regards,
Laxman

Search Discussions

  • Lars George at Jul 4, 2012 at 7:37 am
    Hi Laxman,

    Personally I have not seen many variations - if at all - based on cluster size, apart from the heap settings. What are you proposing should be different for each profile?

    Lars
    On Jul 4, 2012, at 9:15, Laxman wrote:

    Hi Devs,

    How about introducing multiple GC tuning profiles (like STANDALONE, SMALL,
    MEDIUM, LARGE ...) in HBase itself?
    Profiles can be categorized based on cluster size, usage, resources
    available (Memory, CPU).

    Why tuning profiles?
    -> Its tedious & error-prone to find, edit some parameter in script file.
    -> While editing manually and I'm sure we miss one or the other parameters
    in some cluster machine.
    -> These profiles boosts the user's confidence rather tune himself by trial
    and error for each parameter.


    I know I will get the following immediate answers. :-)
    -> it's the responsibility of user to tune GC
    -> it depends on usage
    -> it depends on environment (machine resources available)
    -> we do provide recommendations in hbase book
    -> there are so many resources over web about tuning. Why don't you refer to
    that?
    -> so on...

    I humbly accept all the above reasons but still I wanted to put forward the
    idea of tuning profiles and get your opinion.
    --
    Regards,
    Laxman
  • Laxman at Jul 5, 2012 at 5:01 am

    Personally I have not seen many variations - if at all - based on
    cluster size, apart from the heap settings. What are you proposing
    should be different for each profile?
    Not necessarily. We may have one profile only if we don't have many
    variations. Few I can see [Xms, Xmx, NewSize, MaxNewSize]
    IMHO, user can take advantage of community recommendations (GC collection
    algorithm - HBase documentation, DirectMemorySize - HDFS-3600).

    Also, if possible we can extend this profiles concept to HBase
    configurations as well.
    --
    Regards,
    Laxman
  • Suraj Varma at Jul 5, 2012 at 4:13 pm
    I would like to second Laxman's proposal. Currently, some of the
    "default" hbase configuration is targeted towards newbies so as to
    avoid getting basic questions in the mailing list ... which is ok. I
    think we at least need something like an hbase-prod.xml that lists
    more realistic values for a production like environment - or perhaps
    provide value ranges that are appropriate for a production-like
    environment.

    For instance, default zookeeper.session.timeout of 3 minutes is known
    to be too long ... so, what is a more realistic one for production ...
    and so forth.
    The profiles approach is even better if we see a lot of variations
    depending on cluster / available heap size / etc.

    I don't think anyone should expect blindly using them - but at least
    it gives a much better starting point for tuning the cluster for folks
    taking this into production.
    --Suraj
    On Wed, Jul 4, 2012 at 9:55 PM, Laxman wrote:
    Personally I have not seen many variations - if at all - based on
    cluster size, apart from the heap settings. What are you proposing
    should be different for each profile?
    Not necessarily. We may have one profile only if we don't have many
    variations. Few I can see [Xms, Xmx, NewSize, MaxNewSize]
    IMHO, user can take advantage of community recommendations (GC collection
    algorithm - HBase documentation, DirectMemorySize - HDFS-3600).

    Also, if possible we can extend this profiles concept to HBase
    configurations as well.
    --
    Regards,
    Laxman

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedJul 4, '12 at 7:16a
activeJul 5, '12 at 4:13p
posts4
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase