FAQ
Hi,

I am trying to index documents and when all is complete and optimize is
called I get

IFD [main]: setInfoStream
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
IW 0 [main]: setInfoStream:
dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
autoCommit=false mergePolicy=org
.apache.lucene.index.LogByteSizeMergePolicy@92cb49
mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
ramBufferS
izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 maxFieldLength=10000
index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
_ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numD
ocs=0 numBufDelTerms=0
IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
_ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
_az:c5->_0 _b0:c5->_0 into _b5 [optimize]
IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73]
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
IW 0 [Lucene Merge Thread #73]: now merge
merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
_b5 [optimize]
merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
_az:c5->_0 _b0:c5->_0
IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
_ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
IW 0 [Lucene Merge Thread #73]: hit exception during merge
Exception in thread "Lucene Merge Thread #73"
org.apache.lucene.index.MergePolicy$MergeException:
java.io.FileNotFoundException: /SY
S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
Caused by: java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(FSDirectory.java:552)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:488)
at
org.apache.lucene.index.CompoundFileReader.(SegmentReader.java:321)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP jspMain
SEVERE: Indexing not completebackground merge hit exception: _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
optimize]
java.io.IOException: background merge hit exception: _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
]
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
Source)
at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
Caused by: java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(FSDirectory.java:552)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:488)
at
org.apache.lucene.index.CompoundFileReader.(SegmentReader.java:321)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
IW 0 [main]: now flush at close
IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numD
ocs=0 numBufDelTerms=0
IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: CMS: no more merges pending; now return
IW 0 [main]: now call final commit()
IW 0 [main]: startCommit(): start sizeInBytes=0
IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
95
IW 0 [main]: now sync _37.cfs
IW 0 [main]: hit exception while closing
Exception in thread "main" java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(FSDirectory.java:456)
at
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
at
org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
Source)
at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)



I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying different
combinations of these to see if that helps but was unsuccessful.


Regards,
Rishi
--
View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Michael McCandless at Aug 13, 2009 at 9:49 am
    Is there any chance that two writers are open on this directory? Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM, rishisinghalwrote:
    Hi,

    I am trying to index documents and when all is complete and optimize is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP jspMain
    SEVERE: Indexing not completebackground merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)



    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 13, 2009 at 10:07 am
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory? Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM, rishisinghalwrote:
    Hi,

    I am trying to index documents and when all is complete and optimize is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_160
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception: _61:c245->_
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)



    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Shai Erera at Aug 13, 2009 at 10:09 am
    Is that a local file system, or a network share?
    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal wrote:

    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory? Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    Hi,

    I am trying to index documents and when all is complete and optimize is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)



    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 13, 2009 at 12:00 pm
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory? Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    Hi,

    I am trying to index documents and when all is complete and optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)



    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Shai Erera at Aug 13, 2009 at 12:08 pm
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to system
    crash etc.

    Did you experience that? Can you try to create the index somewhere else, or
    on another drive?

    Shai
    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal wrote:


    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory? Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 13, 2009 at 12:22 pm
    I tried creating the index in different disks but still i see the issue :-(

    I tried to index documents in other disks also and got the same exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to system
    crash etc.

    Did you experience that? Can you try to create the index somewhere else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory?
    Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Aug 13, 2009 at 12:36 pm
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5). Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM, rishisinghalwrote:
    I tried creating the index in different disks but still i see the issue :-(

    I tried to index documents in other disks also and got the same exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to system
    crash etc.

    Did you experience that? Can you try to create the index somewhere else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory?
    Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 13, 2009 at 9:17 pm
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of files this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5). Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM, rishisinghalwrote:
    I tried creating the index in different disks but still i see the issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the directory.
    No this also has rare chances as I am the owner of these files and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory?
    Or,
    something external to Lucene is removing files from the directory.

    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception: _61:c245->_
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Shai Erera at Aug 14, 2009 at 11:09 am
    If it's easily reproducible, can you please provide a test that reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.

    Shai
    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal wrote:


    I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5). Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these files and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1
    _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1
    _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread
    #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
    different
    combinations of these to see if that helps but was unsuccessful.


    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 14, 2009 at 11:37 am
    It is reproducible in my system. If the number of files are small I am not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5). Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1 segments_1.;1 segments_2.;1
    _0.fdt;1
    _0.fdx;1 _37.cfs;1 _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1 _ay.cfs;1 _az.cfs;1
    _b0.cfs;1
    _b2.tis;1 _b3.fdt;1 _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS: launch new thread [Lucene Merge Thread
    #73]
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]: flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Aug 14, 2009 at 11:50 am
    Can you post the full sources for the test that hits the exception on
    your system?

    Mike

    On Fri, Aug 14, 2009 at 7:36 AM, rishisinghalwrote:
    It is reproducible in my system. If the number of files are small I am not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5).  Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1
    _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread
    #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 14, 2009 at 12:13 pm
    This is a big project having lots of source files. The excerpt of the main
    indexer program is:
    FileIndexer indexer = new FileIndexer(props);

    Analyzer defAnalyzer = new StandardAnalyzer();
    FileIndexer.analyzer = defAnalyzer;
    IndexWriter writer = new IndexWriter(dir,FileIndexer.analyzer, true,
    IndexWriter.MaxFieldLength.LIMITED);
    writer.setMaxBufferedDocs(5);
    //writer.setMaxMergeDocs(50);
    writer.setMergeFactor(5);

    long start = new Date().getTime();
    try
    {
    for (int i=0; i<disks.size();i++)
    {
    index(writer, new File(disks.get(i)),filelist, FileIndexer.analyzer);

    }
    writer.setInfoStream(System.out);
    writer.optimize();
    }
    catch(Exception e)
    {
    e.printStackTrace();
    }
    finally
    {

    writer.close();
    }






    Michael McCandless-2 wrote:
    Can you post the full sources for the test that hits the exception on
    your system?

    Mike

    On Fri, Aug 14, 2009 at 7:36 AM, rishisinghalwrote:
    It is reproducible in my system. If the number of files are small I am
    not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run
    after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that
    reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of
    files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5).  Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM,
    rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX
    [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by:
    java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1
    _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread
    #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970750.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Aug 14, 2009 at 12:41 pm
    But, if you can break out just the Lucene indexing part into a
    standalone test, that shows the exception, that can help us isolate
    it.

    At this point it seems likely the issue is something OS/filesystem
    specific, because at one time (during optimize), we see a "file not
    found" exception, yet at another time (running CheckIndex) the same
    file is opened just fine.

    Mike

    On Fri, Aug 14, 2009 at 8:13 AM, rishisinghalwrote:
    This is a big project having lots of source files.  The excerpt of the main
    indexer program  is:
    FileIndexer indexer = new FileIndexer(props);

    Analyzer defAnalyzer = new StandardAnalyzer();
    FileIndexer.analyzer = defAnalyzer;
    IndexWriter writer = new IndexWriter(dir,FileIndexer.analyzer, true,
    IndexWriter.MaxFieldLength.LIMITED);
    writer.setMaxBufferedDocs(5);
    //writer.setMaxMergeDocs(50);
    writer.setMergeFactor(5);

    long start = new Date().getTime();
    try
    {
    for (int i=0; i<disks.size();i++)
    {
    index(writer, new File(disks.get(i)),filelist, FileIndexer.analyzer);

    }
    writer.setInfoStream(System.out);
    writer.optimize();
    }
    catch(Exception e)
    {
    e.printStackTrace();
    }
    finally
    {

    writer.close();
    }






    Michael McCandless-2 wrote:
    Can you post the full sources for the test that hits the exception on
    your system?

    Mike

    On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    It is reproducible in my system. If the number of files are small I am
    not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run
    after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that
    reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of
    files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5).  Something odd is happening at the OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM,
    rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX
    [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by:
    java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information which
    associates it w/ a more native IO problem, like corrupt file due to
    system
    crash etc.

    Did you experience that? Can you try to create the index somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1
    _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread
    #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970750.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 14, 2009 at 12:52 pm
    Okay let me try if i can create a standalone test which reproduces this issue
    and get back.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    But, if you can break out just the Lucene indexing part into a
    standalone test, that shows the exception, that can help us isolate
    it.

    At this point it seems likely the issue is something OS/filesystem
    specific, because at one time (during optimize), we see a "file not
    found" exception, yet at another time (running CheckIndex) the same
    file is opened just fine.

    Mike

    On Fri, Aug 14, 2009 at 8:13 AM, rishisinghalwrote:
    This is a big project having lots of source files.  The excerpt of the
    main
    indexer program  is:
    FileIndexer indexer = new FileIndexer(props);

    Analyzer defAnalyzer = new StandardAnalyzer();
    FileIndexer.analyzer = defAnalyzer;
    IndexWriter writer = new
    IndexWriter(dir,FileIndexer.analyzer, true,
    IndexWriter.MaxFieldLength.LIMITED);
    writer.setMaxBufferedDocs(5);
    //writer.setMaxMergeDocs(50);
    writer.setMergeFactor(5);

    long start = new Date().getTime();
    try
    {
    for (int i=0; i<disks.size();i++)
    {
    index(writer, new File(disks.get(i)),filelist,
    FileIndexer.analyzer);

    }
    writer.setInfoStream(System.out);
    writer.optimize();
    }
    catch(Exception e)
    {
    e.printStackTrace();
    }
    finally
    {

    writer.close();
    }






    Michael McCandless-2 wrote:
    Can you post the full sources for the test that hits the exception on
    your system?

    Mike

    On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    It is reproducible in my system. If the number of files are small I am
    not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run
    after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that
    reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of
    files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram
    available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5).  Something odd is happening at the
    OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM,
    rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX
    [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by:
    java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information
    which
    associates it w/ a more native IO problem, like corrupt file due
    to
    system
    crash etc.

    Did you experience that? Can you try to create the index
    somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these
    files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1
    _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete
    and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false
    numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0
    _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge
    Thread
    #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit
    exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false
    numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0
    _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970750.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24971223.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Rishisinghal at Aug 16, 2009 at 5:56 pm
    hmm i tried to create a small reproducer but was not successful to reproduce
    this. For less number of files my application also works fine. I tried with
    many different individual directories and it runs fine. Only when the
    directory structure is huge i get the io error.

    So i have no idea what is causing this.

    Regards,
    Rishi


    rishisinghal wrote:
    Okay let me try if i can create a standalone test which reproduces this
    issue and get back.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    But, if you can break out just the Lucene indexing part into a
    standalone test, that shows the exception, that can help us isolate
    it.

    At this point it seems likely the issue is something OS/filesystem
    specific, because at one time (during optimize), we see a "file not
    found" exception, yet at another time (running CheckIndex) the same
    file is opened just fine.

    Mike

    On Fri, Aug 14, 2009 at 8:13 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    This is a big project having lots of source files.  The excerpt of the
    main
    indexer program  is:
    FileIndexer indexer = new FileIndexer(props);

    Analyzer defAnalyzer = new StandardAnalyzer();
    FileIndexer.analyzer = defAnalyzer;
    IndexWriter writer = new
    IndexWriter(dir,FileIndexer.analyzer, true,
    IndexWriter.MaxFieldLength.LIMITED);
    writer.setMaxBufferedDocs(5);
    //writer.setMaxMergeDocs(50);
    writer.setMergeFactor(5);

    long start = new Date().getTime();
    try
    {
    for (int i=0; i<disks.size();i++)
    {
    index(writer, new File(disks.get(i)),filelist,
    FileIndexer.analyzer);

    }
    writer.setInfoStream(System.out);
    writer.optimize();
    }
    catch(Exception e)
    {
    e.printStackTrace();
    }
    finally
    {

    writer.close();
    }






    Michael McCandless-2 wrote:
    Can you post the full sources for the test that hits the exception on
    your system?

    Mike

    On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<singhal.rishi@gmail.com>
    wrote:
    It is reproducible in my system. If the number of files are small I am
    not
    able to see the crash.
    can you turn on infoStream on IndexWriter and post the output?
    I am not clear on this. If you are talking of
    writer.setInfoStream(System.out);
    I already have posted the output while raising the issue.

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.
    I am not clear on this too. As I know CheckIndex program has been run
    after
    I get the error and it shows no issues.

    Regards,
    Rishi



    Shai Erera wrote:
    If it's easily reproducible, can you please provide a test that
    reproduces
    it, even if on just your FS? And also, can you turn on infoStream on
    IndexWriter and post the output?

    If CheckIndex succeeds, then I'd compare the output w/ and w/o
    CheckIndex.

    Shai

    On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
    wrote:
    I am running this on OpenVMS V8.2-1 on IA64. For a small number of
    files
    this
    works all fine.

    I checked the resources part and i have enough disk and ram
    available.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    It's very odd that CheckIndex has no trouble opening the segment's
    files, yet when you run optimize the OS reports a "file not found"
    exception (errno 5).  Something odd is happening at the
    OS/filesystem
    level.

    What OS are you running on?

    Can you boil this down to a smallish standalone test that shows the
    problem?

    Mike

    On Thu, Aug 13, 2009 at 8:21 AM,
    rishisinghal<singhal.rishi@gmail.com>
    wrote:
    I tried creating the index in different disks but still i see the
    issue
    :-(

    I tried to index documents in other disks also and got the same
    exception.

    I also tried
    $ java org.apache.lucene.index.CheckIndex
    /SYS$SYSDEVICE/RISHI/melon_1600/
    -segment _61
    NOTE: testing will be more thorough if you run java with
    '-ea:org.apache.lucene...', so assertions are enabled
    Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
    Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX
    [Lucene
    2.4]

    Checking only these segments: _61:
    No problems were detected with this index.

    Regards,
    Rishi


    Shai Erera wrote:
    I noticed the exception is "Caused by:
    java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

    I searched for i/o error (errno:5) and found some information
    which
    associates it w/ a more native IO problem, like corrupt file due
    to
    system
    crash etc.

    Did you experience that? Can you try to create the index
    somewhere
    else,
    or
    on another drive?

    Shai

    On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
    wrote:
    It is a local file system.

    We are using lucene 2.4 and java 1.5

    Regards,
    Rishi


    Shai Erera wrote:
    Is that a local file system, or a network share?

    On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
    wrote:
    Is there any chance that two writers are open on this
    directory?
    No, thats not true.
    something external to Lucene is removing files from the
    directory.
    No this also has rare chances as I am the owner of these
    files
    and
    other
    then me no one can delete the, :-)

    Here are all the files in the index directory:
    segments.gen;1      segments_1.;1       segments_2.;1
    _0.fdt;1
    _0.fdx;1            _37.cfs;1           _4c.cfs;1
    _61.cfs;1
    _9e.cfs;1           _ay.cfs;1           _az.cfs;1
    _b0.cfs;1
    _b2.tis;1           _b3.fdt;1           _b4.fdt;1

    I see _37 present.

    Regards,
    Rishi


    Michael McCandless-2 wrote:
    Is there any chance that two writers are open on this
    directory?
    Or,
    something external to Lucene is removing files from the
    directory.
    It looks like there were at least two missing files (_37

    On Thu, Aug 13, 2009 at 5:19 AM,
    rishisinghal<singhal.rishi@gmail.com
    wrote:
    Hi,

    I am trying to index documents and when all is complete
    and
    optimize
    is
    called I get

    IFD [main]: setInfoStream
    deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
    IW 0 [main]: setInfoStream:
    dir=org.apache.lucene.store.FSDirectory@
    /SYS$SYSDEVICE/RISHI/melon_1600
    autoCommit=false mergePolicy=org
    .apache.lucene.index.LogByteSizeMergePolicy@92cb49
    mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
    ramBufferS
    izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
    maxFieldLength=10000
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: optimize: index now _37:c465->_0
    _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false
    numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0
    _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: add merge to pendingMerges: _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1
    pending]
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   consider merge _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [main]: CMS:     launch new thread [Lucene Merge
    Thread
    #73]
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [Lucene Merge Thread #73]: CMS:   merge thread:
    start
    IW 0 [Lucene Merge Thread #73]: now merge
    merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0
    _b0:c5->_0
    into
    _b5 [optimize]
    merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
    index=_37:c465->_0 _4c:c165->_0 _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0
    _az:c5->_0 _b0:c5->_0
    IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0
    _9e:c485->_0
    _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
    IW 0 [Lucene Merge Thread #73]: hit exception during
    merge
    Exception in thread "Lucene Merge Thread #73"
    org.apache.lucene.index.MergePolicy$MergeException:
    java.io.FileNotFoundException: /SY
    S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    Aug 13, 2009 2:40:56 PM
    org.hp.openvms.lucene.index.FileIndexerJSP
    jspMain
    SEVERE: Indexing not completebackground merge hit
    exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [
    optimize]
    java.io.IOException: background merge hit exception:
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5
    [optimize
    ]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)
    Caused by: java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at
    org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at
    org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at
    org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at
    org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at
    org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
    at
    org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
    IW 0 [main]: now flush at close
    IW 0 [main]:   flush: segment=null docStoreSegment=null
    docStoreOffset=0
    flushDocs=false flushDeletes=true flushDocStores=false
    numD
    ocs=0 numBufDelTerms=0
    IW 0 [main]:   index before flush _37:c465->_0
    _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS: now merge
    IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    IW 0 [main]: CMS:   no more merges pending; now return
    IW 0 [main]: now call final commit()
    IW 0 [main]: startCommit(): start sizeInBytes=0
    IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
    _61:c245->_0
    _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
    changeCount=11
    95
    IW 0 [main]: now sync _37.cfs
    IW 0 [main]: hit exception while closing
    Exception in thread "main" java.io.FileNotFoundException:
    /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error
    (errno:5))
    at java.io.RandomAccessFile.open(Native Method)
    at
    java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
    at
    org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
    at
    org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
    at
    org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
    at
    org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
    at
    org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
    at
    org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
    Source)
    at
    org.hp.openvms.lucene.index.FileIndexer.main(Unknown
    Source)


    I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was
    trying
    different
    combinations of these to see if that helps but was
    unsuccessful.

    Regards,
    Rishi
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail:
    java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail:
    java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
    Sent from the Lucene - Java Users mailing list archive at
    Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context:
    http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970750.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    --
    View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24996067.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedAug 13, '09 at 9:19a
activeAug 16, '09 at 5:56p
posts16
users3
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase