Grokbase Groups HBase dev March 2011
FAQ
Can we add a counter for the number of InternalScanner's to HRegion ?
We decrement this counter when close() is called.

Such counter can be used to gauge the cost of moving the underlying region.

Cheers

Search Discussions

  • Jonathan Gray at Mar 21, 2011 at 11:23 pm
    This is an interesting direction, and definitely file a JIRA as this could be an additional metric in the future, but it's not exactly what I had in mind.

    One of the hardest parts of load balancing based on request count and other dynamic/transient measures is that you can get some pretty pathological conditions where you are always moving stuff around.

    To guard against it, I think we'll need to move to more of a cost-based algorithm that is taking not just the difference in request counts into account but also a baseline "cost" of moving a region. The cost difference in load between two unbalanced servers would have to outweigh the cost associated with moving a region. As you say, looking at the number of live operations to a given region could contribute to the cost of moving that region, but the best measure for that is probably just looking at request count (it's all requests that incur a cost, not just active scanners).

    JG
    -----Original Message-----
    From: Ted Yu
    Sent: Monday, March 21, 2011 3:44 PM
    To: dev@hbase.apache.org
    Subject: gauging cost of region movement

    Can we add a counter for the number of InternalScanner's to HRegion ?
    We decrement this counter when close() is called.

    Such counter can be used to gauge the cost of moving the underlying region.

    Cheers
  • Jonathan Gray at Mar 21, 2011 at 11:26 pm
    Also, using more stable measures of request count will help, such as 30 minute rolling averages.
    -----Original Message-----
    From: Jonathan Gray
    Sent: Monday, March 21, 2011 4:23 PM
    To: dev@hbase.apache.org
    Subject: RE: gauging cost of region movement

    This is an interesting direction, and definitely file a JIRA as this could be an
    additional metric in the future, but it's not exactly what I had in mind.

    One of the hardest parts of load balancing based on request count and other
    dynamic/transient measures is that you can get some pretty pathological
    conditions where you are always moving stuff around.

    To guard against it, I think we'll need to move to more of a cost-based
    algorithm that is taking not just the difference in request counts into account
    but also a baseline "cost" of moving a region. The cost difference in load
    between two unbalanced servers would have to outweigh the cost
    associated with moving a region. As you say, looking at the number of live
    operations to a given region could contribute to the cost of moving that
    region, but the best measure for that is probably just looking at request
    count (it's all requests that incur a cost, not just active scanners).

    JG
    -----Original Message-----
    From: Ted Yu
    Sent: Monday, March 21, 2011 3:44 PM
    To: dev@hbase.apache.org
    Subject: gauging cost of region movement

    Can we add a counter for the number of InternalScanner's to HRegion ?
    We decrement this counter when close() is called.

    Such counter can be used to gauge the cost of moving the underlying region.
    Cheers
  • Ryan Rawson at Mar 21, 2011 at 11:32 pm
    it would make sense to avoid moving regions, so therefore the more
    recently a region was moved, the less likely we should move it.

    you could imagine a hypothetical perfect 'region move cost' function
    that might look like:

    F(r) = timeSinceMoved(r) + size(r) + loadAvg(r)

    The functions should probably be normalized to [0,1], so the range of
    F would be [0,3] with 3 == 'dont move' and 0 == 'move first'.

    The goal is to minimize all the F(r[i]) in the moves.

    -ryan
    On Mon, Mar 21, 2011 at 4:26 PM, Jonathan Gray wrote:
    Also, using more stable measures of request count will help, such as 30 minute rolling averages.
    -----Original Message-----
    From: Jonathan Gray
    Sent: Monday, March 21, 2011 4:23 PM
    To: dev@hbase.apache.org
    Subject: RE: gauging cost of region movement

    This is an interesting direction, and definitely file a JIRA as this could be an
    additional metric in the future, but it's not exactly what I had in mind.

    One of the hardest parts of load balancing based on request count and other
    dynamic/transient measures is that you can get some pretty pathological
    conditions where you are always moving stuff around.

    To guard against it, I think we'll need to move to more of a cost-based
    algorithm that is taking not just the difference in request counts into account
    but also a baseline "cost" of moving a region.  The cost difference in load
    between two unbalanced servers would have to outweigh the cost
    associated with moving a region.  As you say, looking at the number of live
    operations to a given region could contribute to the cost of moving that
    region, but the best measure for that is probably just looking at request
    count (it's all requests that incur a cost, not just active scanners).

    JG
    -----Original Message-----
    From: Ted Yu
    Sent: Monday, March 21, 2011 3:44 PM
    To: dev@hbase.apache.org
    Subject: gauging cost of region movement

    Can we add a counter for the number of InternalScanner's to HRegion ?
    We decrement this counter when close() is called.

    Such counter can be used to gauge the cost of moving the underlying region.
    Cheers
  • Ted Yu at Mar 21, 2011 at 11:36 pm
    I opened HBASE-3679 and pasted comments there.

    Please continue on that JIRA.
    On Mon, Mar 21, 2011 at 4:32 PM, Ryan Rawson wrote:

    it would make sense to avoid moving regions, so therefore the more
    recently a region was moved, the less likely we should move it.

    you could imagine a hypothetical perfect 'region move cost' function
    that might look like:

    F(r) = timeSinceMoved(r) + size(r) + loadAvg(r)

    The functions should probably be normalized to [0,1], so the range of
    F would be [0,3] with 3 == 'dont move' and 0 == 'move first'.

    The goal is to minimize all the F(r[i]) in the moves.

    -ryan
    On Mon, Mar 21, 2011 at 4:26 PM, Jonathan Gray wrote:
    Also, using more stable measures of request count will help, such as 30
    minute rolling averages.
    -----Original Message-----
    From: Jonathan Gray
    Sent: Monday, March 21, 2011 4:23 PM
    To: dev@hbase.apache.org
    Subject: RE: gauging cost of region movement

    This is an interesting direction, and definitely file a JIRA as this
    could be an
    additional metric in the future, but it's not exactly what I had in
    mind.
    One of the hardest parts of load balancing based on request count and
    other
    dynamic/transient measures is that you can get some pretty pathological
    conditions where you are always moving stuff around.

    To guard against it, I think we'll need to move to more of a cost-based
    algorithm that is taking not just the difference in request counts into
    account
    but also a baseline "cost" of moving a region. The cost difference in
    load
    between two unbalanced servers would have to outweigh the cost
    associated with moving a region. As you say, looking at the number of
    live
    operations to a given region could contribute to the cost of moving that
    region, but the best measure for that is probably just looking at
    request
    count (it's all requests that incur a cost, not just active scanners).

    JG
    -----Original Message-----
    From: Ted Yu
    Sent: Monday, March 21, 2011 3:44 PM
    To: dev@hbase.apache.org
    Subject: gauging cost of region movement

    Can we add a counter for the number of InternalScanner's to HRegion ?
    We decrement this counter when close() is called.

    Such counter can be used to gauge the cost of moving the underlying region.
    Cheers

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieshbase, hadoop
postedMar 21, '11 at 10:44p
activeMar 21, '11 at 11:36p
posts5
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase