FAQ
I am getting IOException when I am doing a "Real-time" search, i.e. I am
creating a Index using the Index Writer and also opening the Index using
Index Reader (writer.getReader()) to make sure the document does not exist
prior adding to the Index file.

The code works perfect fine multiple time indexed across couple of GB data,
however, on a TB it fails with the exception listed below.
Any suggestions appreciated. Thanks. Kumar

Code Snippet:

IndexReader reader2 = reader.reopen();//get

if (reader2 != reader) {

reader.close();
reader = reader2;
searcher = new IndexSearcher(reader);
reader2 = null;
}


Exception:

Exception in thread "main" java.io.IOException: directory
'/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
be listed: list() returned null
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
at
org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
at
org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
at
org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
at
org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)

Kumar _/|\_
www.saisk.com
kumar@saisk.com
"making a profound difference with knowledge and creativity..."

Search Discussions

  • Michael McCandless at Dec 31, 2009 at 12:45 pm
    It sounds like you may be running out of file descriptors -- how many
    segments are in your index?

    The reopen logic looks correct (you are closing the old reader). Is
    there anything else that may be holding files open?

    Have you changed any of IW's settings, eg mergeFactor?

    Mike

    On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
    wrote:
    I am getting IOException when I am doing a "Real-time" search, i.e. I am
    creating a Index using the Index Writer and also opening the Index using
    Index Reader (writer.getReader()) to make sure the document does not exist
    prior adding to the Index file.

    The code works perfect fine multiple time indexed across couple of GB data,
    however, on a TB it fails with the exception listed below.
    Any suggestions appreciated. Thanks. Kumar

    Code Snippet:

    IndexReader reader2 = reader.reopen();//get

    if (reader2 != reader) {

    reader.close();
    reader = reader2;
    searcher = new IndexSearcher(reader);
    reader2 = null;
    }


    Exception:

    Exception in thread "main" java.io.IOException: directory
    '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
    be listed: list() returned null
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
    at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
    at
    org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
    at
    org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
    at
    org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
    at
    org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
    at
    org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)

    Kumar    _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Kumaravel Kandasami at Dec 31, 2009 at 5:12 pm
    Thanks Mike.

    I think it is something to do with the merge factor.
    Modified the code to do optimize in the finally block the following error
    message was thrown.

    Code Snippet:
    nameWriter.optimize(); // errors here
    nameWriter.close();
    valueWriter.optimize(); //I am using multiple writers
    simultaneously. Writing to different directory.
    valueWriter.close()


    Index Directory Files:
    _1o.cfs
    _1p.cfs
    _1q.cfs
    _1r.cfs
    _1s.cfs
    _1t.fnm
    segments_1
    segments.gen
    write.lock


    Exception in thread "main" java.io.IOException: background merge hit
    exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize]
    [mergeDocStores]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759)
    at
    com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46)
    at com.hli.vis.index.Main.main(Main.java:39)
    Caused by: java.io.FileNotFoundException:
    /Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(SimpleFSDirectory.java:186)
    at
    org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76)
    at org.apache.lucene.index.FieldsWriter.(SegmentMerger.java:334)
    at
    org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
    Java Result: 1


    Kumar _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."

    On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless wrote:

    It sounds like you may be running out of file descriptors -- how many
    segments are in your index?

    The reopen logic looks correct (you are closing the old reader). Is
    there anything else that may be holding files open?

    Have you changed any of IW's settings, eg mergeFactor?

    Mike

    On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
    wrote:
    I am getting IOException when I am doing a "Real-time" search, i.e. I am
    creating a Index using the Index Writer and also opening the Index using
    Index Reader (writer.getReader()) to make sure the document does not exist
    prior adding to the Index file.

    The code works perfect fine multiple time indexed across couple of GB data,
    however, on a TB it fails with the exception listed below.
    Any suggestions appreciated. Thanks. Kumar

    Code Snippet:

    IndexReader reader2 = reader.reopen();//get

    if (reader2 != reader) {

    reader.close();
    reader = reader2;
    searcher = new IndexSearcher(reader);
    reader2 = null;
    }


    Exception:

    Exception in thread "main" java.io.IOException: directory
    '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
    be listed: list() returned null
    at
    org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
    at
    org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
    at
    org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
    at
    org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
    at
    org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
    at
    org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
    at
    org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
    at
    org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)

    Kumar _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Kumaravel Kandasami at Dec 31, 2009 at 5:35 pm
    Identified the problem.

    reader.close() was not getting called in a specific logic flow.



    Thank You.

    Kumar _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."

    On Thu, Dec 31, 2009 at 11:11 AM, Kumaravel Kandasami wrote:

    Thanks Mike.

    I think it is something to do with the merge factor.
    Modified the code to do optimize in the finally block the following error
    message was thrown.

    Code Snippet:
    nameWriter.optimize(); // errors here
    nameWriter.close();
    valueWriter.optimize(); //I am using multiple writers
    simultaneously. Writing to different directory.
    valueWriter.close()


    Index Directory Files:
    _1o.cfs
    _1p.cfs
    _1q.cfs
    _1r.cfs
    _1s.cfs
    _1t.fnm
    segments_1
    segments.gen
    write.lock


    Exception in thread "main" java.io.IOException: background merge hit
    exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize]
    [mergeDocStores]
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789)
    at
    org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759)
    at
    com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46)
    at com.hli.vis.index.Main.main(Main.java:39)
    Caused by: java.io.FileNotFoundException:
    /Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at
    org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:186)
    at
    org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76)
    at
    org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:82)
    at
    org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334)
    at
    org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153)
    at
    org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029)
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
    at
    org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
    Java Result: 1



    Kumar _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."


    On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless <
    lucene@mikemccandless.com> wrote:
    It sounds like you may be running out of file descriptors -- how many
    segments are in your index?

    The reopen logic looks correct (you are closing the old reader). Is
    there anything else that may be holding files open?

    Have you changed any of IW's settings, eg mergeFactor?

    Mike

    On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami
    wrote:
    I am getting IOException when I am doing a "Real-time" search, i.e. I am
    creating a Index using the Index Writer and also opening the Index using
    Index Reader (writer.getReader()) to make sure the document does not exist
    prior adding to the Index file.

    The code works perfect fine multiple time indexed across couple of GB data,
    however, on a TB it fails with the exception listed below.
    Any suggestions appreciated. Thanks. Kumar

    Code Snippet:

    IndexReader reader2 = reader.reopen();//get

    if (reader2 != reader) {

    reader.close();
    reader = reader2;
    searcher = new IndexSearcher(reader);
    reader2 = null;
    }


    Exception:

    Exception in thread "main" java.io.IOException: directory
    '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
    be listed: list() returned null
    at
    org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
    at
    org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
    at
    org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
    at
    org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
    at
    org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
    at
    org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
    at
    org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
    at
    org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
    at
    org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)

    Kumar _/|\_
    www.saisk.com
    kumar@saisk.com
    "making a profound difference with knowledge and creativity..."
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Jan 1, 2010 at 11:53 am

    On Thu, Dec 31, 2009 at 12:34 PM, Kumaravel Kandasami wrote:
    Identified the problem.

    reader.close() was not getting called in a specific logic flow.
    Phew :) Thanks for bringing closure.

    Mike

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedDec 30, '09 at 8:29p
activeJan 1, '10 at 11:53a
posts5
users2
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase