Hi Sir/Madam,
Below is the question which I posted in this site and as per Mike response, I am sending out an email for having your suggestions.
Anonymous said...
Hi Mike,
This is Ravi,We are using Lucene-2.0 in our email archiving product for indexing/searching the documents.
We have been facing a critical problem which affecting the production on customer sites, the problem is while optimization taking place on larger indices of size > 2 GB, the indexer threads getting into blocked state, since index writer opened for optimization purpose is never getting released back for ongoing indexing. Here i am giving you thread dump of blocked indexer threads and optimizer thread.
I appreciate your help and suggestion in this regard.
July 19, 2011 12:50 PM
Mike McCandless said...
Hi Ravi,
Can you bring this question to Lucene's user's list? (ie, send an email to [email protected]).
July 19, 2011 2:23 PM
Anonymous said...
Definitely Mike, I will be sending out an email with full details and thread dumps which i have.
Index Sizes:
All the indices size is more than 1 GB, as per my observations in logs, while optimization taking place on below indices, Index closeWriter operation taking much time, its in between 1 minute to 2 minutes as part
Of basic optimization step, at the same time Indexer threads going into blocked state, since the Indexer threads will not get an access on the same index on which Optimizer threads working, which leads to a race condition. So, that it slowing down indexing process, on the other hand it will lead to store rate issues.
I also looked at QCCR2B5573, which is exactly similar to this new case, thread dumps are similar, NPI has commented 5573 case as an enhancement/design change issue.
- locked <0x00002aaabc869700> (a org.apache.lucene.index.SerialMergeScheduler)
[[email protected] ~]# du --si /index/*
392M /index/TP_0000000000000000481
1.2G /index/TP_0000000000000000482
1.2G /index/TP_0000000000000000483
1.2G /index/TP_0000000000000000484
1.2G /index/TP_0000000000000000485
1.1G /index/TP_0000000000000000486
1.2G /index/TP_0000000000000000487
1.2G /index/TP_0000000000000000488
1.2G /index/TP_0000000000000000489
1.1G /index/TP_0000000000000000490
1.2G /index/TP_0000000000000000491
1.3G /index/TP_0000000000000000492
1.2G /index/TP_0000000000000000493
1.2G /index/TP_0000000000000000494
1.1G /index/TP_0000000000000000495
1.2G /index/TP_0000000000000000496
1.1G /index/TP_0000000000000000497
2.2G /index/TP_0000000000000000498
1.3G /index/TP_0000000000000000499
1.3G /index/TP_0000000000000000500
1.2G /index/TP_0000000000000000501
1.2G /index/TP_0000000000000000502
1.2G /index/TP_0000000000000000503
1.1G /index/TP_0000000000000000504
381M /index/TP_0000000000000000505
[[email protected] ~]#
"Indexer_1" daemon prio=10 tid=0x000000005e09ec00 nid=0x2527 waiting for monitor entry [0x000000004743a000..0x000000004743ab10]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:33)
- waiting to lock <0x00002aaabc869700> (a org.apache.lucene.index.SerialMergeScheduler)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2133)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2128)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2124)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2822)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2807)
at com.hp.platform.store.indexmanager.Index.closeWriter(Index.java:968)
at com.hp.platform.store.indexer.IndexerThread.closeIndexes(IndexerThread.java:304)
at com.hp.platform.store.indexer.IndexerThread.run(IndexerThread.java:232)
at com.hp.platform.core.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:188)
"Indexer_0" daemon prio=10 tid=0x000000005fd52c00 nid=0x2526 waiting for monitor entry [0x0000000047339000..0x0000000047339c90]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:33)
- waiting to lock <0x00002aaabc869700> (a org.apache.lucene.index.SerialMergeScheduler)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2133)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2128)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2124)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2822)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2807)
at com.hp.platform.store.indexmanager.Index.closeWriter(Index.java:968)
at com.hp.platform.store.indexer.IndexerThread.closeIndexes(IndexerThread.java:304)
at com.hp.platform.store.indexer.IndexerThread.run(IndexerThread.java:232)
at com.hp.platform.core.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:188)
"OptimizerThread" daemon prio=10 tid=0x000000005e6f1800 nid=0x2525 runnable [0x0000000047238000..0x0000000047238c10]
java.lang.Thread.State: RUNNABLE
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:554)
- locked <0x00002aaabc86a998> (a org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor)
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:80)
at org.apache.lucene.index.SegmentTermPositions.readDeltaPosition(SegmentTermPositions.java:81)
at org.apache.lucene.index.SegmentTermPositions.nextPosition(SegmentTermPositions.java:76)
at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:567)
at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:494)
at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:463)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:423)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:142)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3571)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3266)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:36)
- locked <0x00002aaabc869700> (a org.apache.lucene.index.SerialMergeScheduler)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2133)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2065)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2013)
at com.hp.platform.store.indexmanager.Index.runBasicOptimization(Index.java:674)
at com.hp.platform.store.indexmanager.OptimizerThread.optimize(OptimizerThread.java:152)
at com.hp.platform.store.indexmanager.OptimizerThread.run(OptimizerThread.java:56)
at com.hp.platform.core.utils.threads.ThreadPool$Pooled_thread.run(ThreadPool.java:188)
Regards
Ravi
