I need to load 1 million queue messages into a hbase table in 30 mins.
As "HBase: The Definitive Guide" suggests, I use Client API, flushCommits().
I launched, say, 20 threads, each thread has its own queue connection and hbase instance, which read msg from queue and insert it into hbase. At the end of thread life, it will call the hbaseTable.flushCommits().
It seems working fine, except for:
1. Each flushCommit action takes quite a long time.
2. Occasionally, flushCommit cause WrongRegionException.
Can someone please share the best practice in this situation? Especially, should each thread call flushCommit() or only the main thread call it?