[HBase-user] data loss after killing RS

What do you mean by kill the region server? kill -9, unplug the power, cut the network?

When you flush commits, it means the data made it to the RS. Without WAL, it doesn't mean it made it to HDFS. It can be only in memory (memstore). When you kill the process abruptly then, it will lose the data.

A normal shutdown flushes the memstore to HDFS, I think.


Friso

________________________________________
Van: Qing Yan [[email protected]]
Verzonden: vrijdag 27 mei 2011 4:43
Aan: [email protected]
Onderwerp: data loss after killing RS

Hello,
I found something strange, here is the test case:
1) Process A insert data into a particular hbase region, WAL off, AutoFlush
off
2) Process A issues htable.flushCommits(), no exception thrown, write down
the row key.
4) Kill the region server manually
5) Process B query the row key, but can't find it no matter how many times
it retries. (In the meantime via
hbase UI, the region get reassigned)
Is this expected? I am using the lastest Cloudera build.

Thank you.

2 things:

- If you kill -9 before the flush happened, data isn't persisted.
It's an async operation BTW, even if you call flush and the call
returns it doesn't mean the data is already on disk.
- About step 5, what happens when you wait for the region to be
reassigned? The default zk timeout is pretty long so that people that
didn't tune their JVM settings don't run into region servers dying all
the time.

J-D

Ok, thanks for the explaination. so data loss is normal in this case.
Yeah , I did a "kill -9". I did wait till the RS get reassigned and
actually let process B keep retring over the night ..

Is WAL the only way to guarantee data safety in hbase? We want high insert
rate though.
Is there a middle ground? e.g. a sync operation to flush RS to HDFS will be
perfect!


>

Well, I realized myself RS flush to HDFS is not designed to do incremental
changes. So there is no way around of WAL? man..just wish it can run a bit
faster:-P

You might want to look into bulk loading.

-Joey
insert
be

Or actually flush the table rather than just flushing commits:
http://archive.cloudera.com/cdh/3/hbase-0.90.1-cdh3u0/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#flush(byte[])

-Todd

Ok, but our app has online/realtime processing requirements. My
understanding bulk importing requires M/R job and is only good for
batch processing?

The Javadoc says HBaseAdmin flush is an async operation. How do I get
the confirmation whether it succeeded or not?

Have you looked at deferred flushing? Its an attribute you set on
your table. You then say how often to run sync using
'hbase.regionserver.optionallogflushinterval'. Default is sync every
second.

St.Ack

Hello Stack,

That parameter is for WAL right? I am trying to find a way to achieve
reliable persistency in HBase without the WAL slowness, but it looks like
impossible ..

You specify it when you set the schema for your table. Try it.

Yes, syncing each write is expensive. There is a grouping effect
going on in the dfsclient which makes it so we are not sync'ing each
write but it could be the case if for example your edits are large?
What size are they?

St.Ack