[HBase-dev] initial experience with HBase 0.90.1 rc0

FYI I use cdh3b2. I put hadoop jar from cdh3b2 into $HBASE_HOME/lib

So, .META. is not online? What happens if you use shell at this time.

Your attachement did not come across Ted. Mind postbin'ing it?

St.Ack

.META. went offline during second flow attempt.

The time out I mentioned happened for 1st and 3rd attempts. HBase was
restarted before the 1st and 3rd attempts.

Here is jstack:
http://pastebin.com/EHMSvsRt

You don't have both the old and the new hbase jars in there do you?

-ryan

hbase-0.90.0-tests.jar and hbase-0.90.1.jar co-exist
Would this be a problem ?

What do you get when you:

ls lib/hbase*

I'm going to guess there is hbase-0.90.0.jar there

hbase/hbase-0.90.1.jar leads lib/hbase-0.90.0.jar in the classpath.
I wonder
1. why hbase jar is placed in two directories - 0.20.6 didn't use such
structure
2. what from lib/hbase-0.90.0.jar could be picked up and why there wasn't
exception in server log

I think a JIRA should be filed for item 2 above - bail out when the two
hbase jars from $HBASE_HOME and $HBASE_HOME/lib are of different versions.

Cheers

As I suspected.

It's a byproduct of our maven assembly process. The process could be
fixed. I wouldn't mind. I don't support runtime checking of jars,
there is such thing as too much tests, and this is an example of it.
The check would then need a test, etc, etc.

At SU we use new directories for each upgrade, copying the config
over. With the lack of -default.xml this is easier than ever (just
copy everything in conf/). With symlink switchover it makes roll
forward/back as simple as doing a symlink switchover or back. I have
to recommend this to everyone who doesnt have a management scheme.

Can someone comment on my second question ?
Thanks

It's a standard linking issue, you get one class from one version
another from another, they are mostly compatible in terms of
signatures (hence no exceptions) but are subtly incompatible in
different ways. In the stack trace you posted, the handlers were
blocked in:

at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:382)

and the thread:

"regionserver60020.cacheFlusher" daemon prio=10 tid=0x00002aaabc21e000
nid=0x7717 waiting for monitor entry [0x0000000000000000]
java.lang.Thread.State: BLOCKED (on object monitor)

was idle.

The cache flusher thread should be flushing, and yet it's doing
nothing. This also happens to be one of the classes that were
changed.

Thanks for the explanation.
Assuming the mixed class loading is static, why did this situation develop
after 40 minutes of heavy load :-(

You didn't hit global memstore pressure until 40 minutes of load.

-Todd

I had 3 consecutive successful runs processing 200GB data for each run
before hitting timeout problem in the 4th run.

The 5th run couldn't proceed because master complained:

2011-02-13 16:11:45,173 FATAL org.apache.hadoop.hbase.master.HMaster: Failed
assignment of regions to
serverName=sjc1-hadoop6.sjc1.carrieriq.com,60020,1297518996557,
load=(requests=0, regions=231, usedHeap=3535, maxHeap=3983)

but sjc1-hadoop6.sjc1 claimed:
2011-02-13 16:13:32,258 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: No master found, will
retry

Here is stack trace for sjc1-hadoop6.sjc1:
http://pastebin.com/X8zWLXqu

I didn't have chance to capture master stack trace as master exited after
that.

I also attach master and region server log on sjc1-hadoop6.sjc1 - pardon me
for including individual email addresses as attachments wouldn't go through
hbase.apache.org

every handler thread, and every reader and also the accept thread are
all blocked on flushing memstore. The handlers get blocked, then the
readers also have a finite handoff queue and they are blocked and also
the accept.

But why isnt memstore flushing? Do you have regionserver stats? ie:
how much memstore global ram used? That is found on the main page of
the regionserver http service, also found in ganglia/file stats.

I havent looked at the logs yet, I'm off to lunch now.

-ryan

I disabled MSLAB.
My flow still couldn't make much progress.

MSLAB is already disabled by default in 0.90.1. If it's not, I screwed up
the commits and we should fix that.

-Todd

BTW
The timeout (when calling flushCommits) happened midnight, so I didn't
capture jstack.

In hadoop1 region server log, I see this around time of timeout in 4th run:

2011-02-13 08:25:01,015 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2011-02-13 08:25:01,016 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call flushRegion(REGION => {NAME =>
'NIGHTLYDEVGRIDSGRIDSQL-THREEGPPSPEECHCALLS-1297583809865,2>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&T,1297583814638.8cb772d452dee232306dfab0b472ec9a.',
STARTKEY => '2>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&U\xF6\xB582>&T',
ENDKEY =>
'2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDFhVz2\xC1\xA3\xDD',
ENCODED => 8cb772d452dee232306dfab0b472ec9a, TABLE => {{NAME =>
'NIGHTLYDEVGRIDSGRIDSQL-THREEGPPSPEECHCALLS-1297583809865', FAMILIES =>
[{NAME => 'd', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
'2', COMPRESSION => 'GZ', TTL => '31536000', BLOCKSIZE => '65536', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}, {NAME => 'i', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '2', COMPRESSION => 'GZ', TTL =>
'31536000', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}, {NAME => 'v', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
VERSIONS => '2', COMPRESSION => 'GZ', TTL => '31536000', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}) from
10.202.50.76:62489: output error
2011-02-13 08:25:01,020 WARN org.apache.hadoop.ipc.HBaseServer: PRI IPC
Server handler 3 on 60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1339)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)

Here is partial config I used:
http://pastebin.com/1Dpbb2LA

I verified that there is no hbase-0.90.1.jar in lib dir.

Thanks