[HBase-dev] HBase balancer policy issue

Ted Yu

at May 20, 2011 at 3:34 pm

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk, at
least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write operations
would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in 0.90
branch.

Refer to
http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html for
details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations hit
RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

reply | permalink

Schubert Zhang

at May 28, 2011 at 4:58 pm

⇧

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk, at
least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write operations
would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in 0.90
branch.

Refer to
http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.html for
details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations hit
RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

reply | permalink

Anty

at Jun 1, 2011 at 2:44 am

⇧

When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk, at
least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write operations
would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in 0.90
branch.

Refer to
http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor
details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com>
wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations hit
RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Stack

at Jun 1, 2011 at 3:03 am

⇧

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk, at
least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write operations
would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in 0.90
branch.

Refer to
http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor
details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com>
wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations hit
RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Schubert Zhang

at Jun 1, 2011 at 4:14 am

⇧

We have encounter following issue:

1. A region have many files, the compacting takes long time.

2. But the balancer (default 5 minutes) close and move this region to
another server.

3. Then, the compacting start again.

4. Then, then balancer close and move it to another server.

Thus, the compacting cannot complete.

Now, we set the balancer interval to 30 minutes to remission this issue.

Schubert

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk,

at

least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com>
wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Anty

at Jun 1, 2011 at 6:26 am

⇧

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

Yes, that's exactly what i want to convey.
Can this consideration be included in balancer?

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk,

at

least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com>
wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Ted Yu

at Jun 1, 2011 at 4:33 pm

⇧

I logged https://issues.apache.org/jira/browse/HBASE-3943
This seems to be a popular request.

I will wait for other developers' comment first.

On Tue, May 31, 2011 at 11:26 PM, Anty wrote:

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

Yes, that's exactly what i want to convey.
Can this consideration be included in balancer?

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang <zsongbo@gmail.com>

wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk,

at

least).
If the tables are created with pre-split regions, the regions would

be

round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young regions

to

other

servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com

wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number

of

regions

and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Stack

at Jun 1, 2011 at 5:25 pm

⇧

Interesting. I suppose if the balancer remembered what it had
recently moved and let some time elapse before it moved the recently
moved, would that do?
St.Ack

On Tue, May 31, 2011 at 11:26 PM, Anty wrote:

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

Yes, that's exactly what i want to convey.
Can this consideration be included in balancer?

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang <zsongbo@gmail.com>

wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in trunk,

at

least).
If the tables are created with pre-split regions, the regions would be
round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young regions to other
servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <zsongbo@gmail.com>
wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number of regions
and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Anty

at Jun 2, 2011 at 8:19 am

⇧

if region A is doing compaction , load balancer ask region server to close
region A
if set hbase.hstore.close.check.interval = 0,
the region A can't be closed until the completion of compaction,
during this period, the region A can also accept mutation request?
Am I right?
Does this have bad affect on load balancer?

On Thu, Jun 2, 2011 at 1:25 AM, Stack wrote:

Interesting. I suppose if the balancer remembered what it had
recently moved and let some time elapse before it moved the recently
moved, would that do?
St.Ack

On Tue, May 31, 2011 at 11:26 PM, Anty wrote:

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

Yes, that's exactly what i want to convey.
Can this consideration be included in balancer?

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status of
regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang <zsongbo@gmail.com>

wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in

trunk,

at

least).
If the tables are created with pre-split regions, the regions would

be

round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young regions

to

other

servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <

zsongbo@gmail.com>

wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the number

of

regions

and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

Schubert Zhang

at Jun 2, 2011 at 3:42 pm

⇧

Thanks Ted.

On Thu, Jun 2, 2011 at 4:19 PM, Anty wrote:

if region A is doing compaction , load balancer ask region server to close
region A
if set hbase.hstore.close.check.interval = 0,
the region A can't be closed until the completion of compaction,
during this period, the region A can also accept mutation request?
Am I right?
Does this have bad affect on load balancer?

On Thu, Jun 2, 2011 at 1:25 AM, Stack wrote:

Interesting. I suppose if the balancer remembered what it had
recently moved and let some time elapse before it moved the recently
moved, would that do?
St.Ack

On Tue, May 31, 2011 at 11:26 PM, Anty wrote:

On Wed, Jun 1, 2011 at 11:02 AM, Stack wrote:

If compacting, don't we interrupt it so we can close and move the
region. You are worried about the compacting work done so far -- you
don't want to lose it? So you are suggesting that a region should be
able to say "No, not now! I'm busy?" (We'd need to distingush between
a balancer 'move' and a move or close for any other region).

Yes, that's exactly what i want to convey.
Can this consideration be included in balancer?

St.Ack

On Tue, May 31, 2011 at 7:44 PM, Anty wrote:
When doing balance, Can we take into account the compaction status

of

regions.
Currently, even the region is doing compaction, it can also be
interrupted
to response to reassign.

On Sun, May 29, 2011 at 12:57 AM, Schubert Zhang <zsongbo@gmail.com

wrote:

Thanks, I think it is HBASE-3373

On Fri, May 20, 2011 at 11:33 PM, Ted Yu <yuzhihong@gmail.com>

wrote:

I think your request is described in
https://issues.apache.org/jira/browse/HBASE-3373

Practically speaking, the scenario below can hardly occur (in

trunk,

at

least).
If the tables are created with pre-split regions, the regions

would

be

round-robin distributed.
If the tables are created with single region, subsequent write

operations

would cause the region split. Balancer would offload young

regions

to

other

servers - see HBASE-3609
<https://issues.apache.org/jira/browse/HBASE-3609>which is not

in

0.90

branch.

Refer to

http://zhihongyu.blogspot.com/2011/04/load-balancer-in-hbase-090.htmlfor

details.

Cheers

On Fri, May 20, 2011 at 12:25 AM, Schubert Zhang <

zsongbo@gmail.com>

wrote:

I have a question about HBase balancer.

In release 0.90.x, it seems the balancer only regards the

number

of

regions

and balance these regions into every regionserver.

If we have two tables (A and B) now, each have 100 regions.
Then, a extreme situation is:

RegionsServer1: 100 regions, which all belong to table A
RegionsServer2: 100 regions, which all belong to table B

If my application access table B heavy, the almost all

opetations

hit

RegionsServer2, it is not balance.

I have a idea about the balance policy:
(1) Firstly balance for each table
(2) Then, overall balance.

Schubert

--
Best Regards
Anty Rao

reply | permalink

group	dev @
categories	hbase, hadoop
posted	May 20, '11 at 7:26a
active	Jun 2, '11 at 3:42p
posts	11
users	4
website	hbase.apache.org

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[HBase-dev] HBase balancer policy issue

Search Discussions

Related Discussions

4 users in discussion