[HBase-dev] Loaded Coprocessor

Hi,

Path-1 add another mertic for HRegion to count request made to region.

Path-2 add another command to hbase shell to grab all regions, sort by
requests from Path-1 and move in round-robin algorithm to servers


Path-1: http://paste.org/pastebin/view/28526
Path-2: http://paste.org/pastebin/view/28528


PS. I'm not a java nor ruby coder but this code working for us and pass
tests :)

Regards,
SB

Path-2 had small bug that balancer wasn't turn on again, so this is new
url: http://paste.org/pastebin/view/28530

In HRegion.java:
final AtomicInteger requestsCount = new AtomicInteger(0);

AtomicLong should be used.

then why hbase have this?

in HRegionServer.java:206
// Request counter
private volatile AtomicInteger requestCount = new AtomicInteger();

path-1 with AtomicLong:
http://paste.org/pastebin/view/28532

Hey this is pretty cool, I've been wanting something like that for a
while. It's a little bit rough but the idea is there. Ultimately, I
hope HBase will be able to provide fine-grained metrics on a
per-region basis and use that to do load balancing.

The problem right now is that load balancing is very costly for
clients, because it takes too long to move a region around in a real
cluster. The region stays down for several seconds (!). This is
really disruptive for high-throughput low-latency user-facing
applications. So until we make the region migration process more
seamless, we can't really have a very aggressive / proactive load
balancer.

The other day I suggested an idea to Stack to change the region
migration process with virtually zero downtime. It basically involves
telling the source region server where the region is going to land,
and telling the destination region server to prepare to receive the
region. The source RS would do a first flush and remember the point
at which it is (there's a generation ID or something already, whatever
is needed for ACID). The source RS would send an RPC to the
destination RS to tell it to start loading whatever was flushed and
then it would replicate all the edits to the destination RS. Once
both RS are in sync, the source RS would block requests to the region
(by locking it), tell the destination RS about it, and after getting
the final ACK from it, would update META and send a special NSRE to
all the clients of the blocked requests. The special NSRE would
basically just be like a normal NSRE but it would also say "hint: I
think the region is now on that RS". From the clients' point of view,
the region downtime would be pretty minimal (almost unnoticeable).
Also, this scheme allows for opportunities such as warming up the
block cache in the destination RS before it starts serving.

Benoît:

Stick the below up in an issue. Its good stuff.

St.Ack

JIRA created: https://issues.apache.org/jira/browse/HBASE-3507

This is a good suggestion.
We also need to consider the situation where source RS is under heavy load,
facing potential GC pause. In this case we shouldn't expect source RS to
carry on too much computation.

Whenever things fail unexpectedly, we can always fall back to the
current way of migrating regions (expensive for clients but easier for
the server).

Note that the problem is also valid if the destination RS is having
load / GC issues. But hopefully once we expose proper load metrics,
the master will be able to make good load balancing decisions. If the
master has information such as QPS per region, latency per region,
number of GC cycles per GC type, then it can get a pretty clear
picture of the load of each RS and try to even it out.

About 'number of GC cycles per GC type', is there API to get this
information ?

Regards

As far as I know this is only exposed through JMX. I'm sure there is
a way to use some horrible APIs in the JDK to "efficiently" access JMX
stuff from the local JVM. If there's another way than JMX, I'd love
to know.

The way I collect HBase metrics for OpenTSDB is to use the Sun private
API to attach to the JVM and get a JXM URL, even if the VM wasn't
configured to expose any JMX thing on a port. I think under the hood
it starts a thread in the remote JVM to handle RMI calls that probably
go over the loopback interface. This is pretty much how jconsole and
friends work, BTW. Pretty horrible stuff. :(

This means we need horrible stuff to write good stuff :-)

http://download.oracle.com/javase/1.5.0/docs/api/java/lang/management/ManagementFactory.html#getGarbageCollectorMXBeans()
<http://download.oracle.com/javase/1.5.0/docs/api/java/lang/management/ManagementFactory.html#getGarbageCollectorMXBeans()>exposes
some of the info

Unfortunately you can't get at any of the really juicy details like CMS FLS
statistics :(

-Todd

Do you have to resort to reading back your own GC log to figure this
stuff out? :(

This approach would introduce dependency on the underlying JVM - vendor,
version, etc because the log format can change across JVM versions.

Yeah i know move take a while but we have small cluster(3 machines) and
when few hot regions land in one machine then we had about
4000/1000/1000 requests on machines after a rebalancing we have now
about 1900/2200/1900 so its worked :)

maybe someone can write tool to take few regions from overloaded
regionserver and put on the others, but i have no skills to write it in
ruby ;)

What version of HBase do you use ?
Looking at the line numbers, you are using version older than 0.90

Also, incrementing counter in startRegionOperation() doesn't consider the
following call (in 0.90 RC3):
public Integer obtainRowLock(final byte [] row) throws IOException {
startRegionOperation();

Regards

its trunk version:

Path: .
URL: https://svn.apache.org/repos/asf/hbase/trunk
Repository Root: https://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1067865
Node Kind: directory
Schedule: normal
Last Changed Author: stack
Last Changed Rev: 1067598
Last Changed Date: 2011-02-06 06:46:39 +0100 (Sun, 06 Feb 2011)

Sebastian:

I was going to say please make an issue with this good stuff but I
notice you just started in.

Good on you Sebastian. Thanks,
St.Ack

Hi JS.
loaded? Preferebly using the shell...

No. Not yet. But that's a reasonable requirement. You can open a jira
for it. Right now you have to check HTableDescriptor (for table
coprocessors) or configurations (for system-wide cp) to see loaded
coprocessors.
hbase system?

Yes, and no. You can dynamically load a table-based coprocessor: 1)
disable a table; 2) set an HTableDecriptor property; 3) enable the
table. Please read ``Load from table attribute'' section of
http://hbaseblog.com/2010/11/30/hbase-coprocessors/. But for system cp,
you cannot do it right now.

It's absolutely doable to have an easier way to dynamically load/unload
coprocessor. But it might relate to security issue: you don't want
anyone from your cluster to easily load/unload a customized cp without
any limitation.

There is still rooms to improve coprocessor loading/unloading mechanism.
It will depend on how users want to use it. Welcome to provide your
feedback and contribution.

-Mingjie

Currently coprocessors are loaded statically via configuration or dynamically upon table open if there are any table attributes that specify it (attribute name begins with "Coprocessor"). IIRC the javadoc on the coprocessors package contains detail on this. In short:

HTableDescriptor htd;
// ...
htd.setValue("Coprocessor$foo", "hdfs://namenode:port/path/to/coprocessor.jar");

A table descriptor can be updated if such specifications are not made a table create time. See javadoc for HBaseAdmin.

However there is currently no mechanism for triggering additional loads on tables already open or loading at arbitrary times. There hasn't been a clearly articulated need for this yet.

Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back.
- Piet Hein (via Tom White)


--- On Mon, 2/7/11, js_84@email.de wrote:

Hi,
thanks for your feedback!

Just a follow up question;
According to the TableDescriptor
{NAME => 'test3', Coprocessor$2 => '/Users/hbase/CP.jar:Coprocessor.PreGet:12', Coprocessor$1 => '/Users/hbase/CP.jar:Coprocessor.PrePut:10',
FAMILIES => [{NAME => 'cf', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}

the coprocessors is specified, but actually neither of them is executed when executing a put; So I am not sure whether they are actually loaded (what happens if if jar file is not found? -> this should trigger at least a warning).

Any ideas/pointers why this fails (HBAse Version 0.91.0-SNAPSHOT, r1065918)?

package Coprocessor;
public class PrePut extends BaseRegionObserverCoprocessor {
@Override
public void prePut( RegionCoprocessorEnvironment e, Map<byte[],
List<KeyValue>> familyMap, boolean writeToWAL) throws IOException {

//bypass normal system call
e.bypass();
}
}


Thanks again!
JS


-----Ursprüngliche Nachricht-----
Von: "Andrew Purtell" <apurtell@apache.org>
Gesendet: 07.02.2011 23:42:35
An: dev@hbase.apache.org
Betreff: Re: Loaded Coprocessor
Currently coprocessors are loaded statically via configuration or dynamically upon table open if there are any table attributes that specify it (attribute name begins with "Coprocessor"). IIRC the javadoc on the coprocessors package contains detail on this. In short:

HTableDescriptor htd;
// ...
htd.setValue("Coprocessor$foo", "hdfs://namenode:port/path/to/coprocessor.jar");

A table descriptor can be updated if such specifications are not made a table create time. See javadoc for HBaseAdmin.

However there is currently no mechanism for triggering additional loads on tables already open or loading at arbitrary times. There hasn't been a clearly articulated need for this yet.

Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back.
- Piet Hein (via Tom White)


--- On Mon, 2/7/11, js_84@email.de wrote:

Hi JS.

Here are some hints to help you debug:
- make sure you have the jar copied to all the region servers if you
want it to sit on local fs, or you can put it to hdfs so all RS can
access it without copying arount.
- remember to disable, and enable the table to make sure the attribute
be loaded
- if you still cannot see it working, please post your RS log after the
time of table enabled.

Loading cp classes from file system lacks of testing, so there might be
some issues. I added a jira:
https://issues.apache.org/jira/browse/HBASE-3516 to add test cases back
for class loading.

Another workaround is to use configuration to test your coprocessor.

This has been tested a lot among us. The only problem is that it would
be applied to all your tables. You can use this method if there is an
issue for cp class loading from jar files.

Thanks,
Mingjie