FAQ
Hi,

I have a single IndexWriter object which I use to update
the index. After each update, I'd like to query the index
using IndexReader and IndexSearcher objects.

When I try to do that I get java.io.FileNotFoundException:
/tmp/lucene/_32.fdx (Too many open files).

lsof -p says that there are many open file handles to the
same index file name, all but the last with a (deleted) marker.

I think I close everything properly.

Here's the code for the test case which reproduces the issue:

Directory ldir = new SimpleFSDirectory(new File("/tmp/lucene"));

IndexWriter writer = new IndexWriter(ldir, new
StandardAnalyzer(Version.LUCENE_30), IndexWriter.MaxFieldLength.LIMITED);
IndexReader reader = IndexReader.open(ldir);

IndexSearcher searcher = new IndexSearcher(reader);

for (int i = 0; i < 1000; i++) {
Document doc = new Document();
doc.add(new Field("uri", "http://example.org/", Field.Store.YES, Field.Index.NO));
writer.addDocument(doc);
writer.commit();

reader.reopen();
searcher.close();
searcher = new IndexSearcher(reader);

// do something here

Term term = new Term("uri", "http://example.org/");
writer.deleteDocuments(term);
writer.commit();
}

Any idea what's going wrong?

Cheers,
Andreas.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Ahmet Arslan at Jan 8, 2011 at 4:47 pm

    --- On Sat, 1/8/11, Andreas Harth wrote:

    From: Andreas Harth <andreas@harth.org>
    Subject: Frequent updates lead to "Too many open files"
    To: java-user@lucene.apache.org
    Date: Saturday, January 8, 2011, 6:30 PM
    Hi,

    I have a single IndexWriter object which I use to update
    the index.  After each update, I'd like to query the
    index
    using IndexReader and IndexSearcher objects.

    When I try to do that I get java.io.FileNotFoundException:
    /tmp/lucene/_32.fdx (Too many open files).

    lsof -p says that there are many open file handles to the
    same index file name, all but the last with a (deleted)
    marker.

    I think I close everything properly.

    Here's the code for the test case which reproduces the
    issue:

    Directory ldir = new SimpleFSDirectory(new
    File("/tmp/lucene"));

    IndexWriter writer = new IndexWriter(ldir, new
    StandardAnalyzer(Version.LUCENE_30),
    IndexWriter.MaxFieldLength.LIMITED);
    IndexReader reader = IndexReader.open(ldir);

    IndexSearcher searcher = new IndexSearcher(reader);

    for (int i = 0; i < 1000; i++) {
    Document doc = new Document();
    doc.add(new Field("uri", "http://example.org/", Field.Store.YES,
    Field.Index.NO));
    writer.addDocument(doc);
    writer.commit();

    reader.reopen();
    searcher.close();
    searcher = new IndexSearcher(reader);

    // do something here

    Term term = new Term("uri", "http://example.org/");
    writer.deleteDocuments(term);
    writer.commit();
    }

    Any idea what's going wrong?
    Doing commit operation at the end of the loop can solve your problem. It is not a good idea to commit after every single addition.




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Ian Lea at Jan 8, 2011 at 5:14 pm
    You also need to read the javadocs for reopen(), and the sample code
    there. And it would be worth reading up on lucene's near real-time
    (NRT) features.


    --
    Ian.

    On Sat, Jan 8, 2011 at 4:46 PM, Ahmet Arslan wrote:

    --- On Sat, 1/8/11, Andreas Harth wrote:
    From: Andreas Harth <andreas@harth.org>
    Subject: Frequent updates lead to "Too many open files"
    To: java-user@lucene.apache.org
    Date: Saturday, January 8, 2011, 6:30 PM
    Hi,

    I have a single IndexWriter object which I use to update
    the index.  After each update, I'd like to query the
    index
    using IndexReader and IndexSearcher objects.

    When I try to do that I get java.io.FileNotFoundException:
    /tmp/lucene/_32.fdx (Too many open files).

    lsof -p says that there are many open file handles to the
    same index file name, all but the last with a (deleted)
    marker.

    I think I close everything properly.

    Here's the code for the test case which reproduces the
    issue:

    Directory ldir = new SimpleFSDirectory(new
    File("/tmp/lucene"));

    IndexWriter writer = new IndexWriter(ldir, new
    StandardAnalyzer(Version.LUCENE_30),
    IndexWriter.MaxFieldLength.LIMITED);
    IndexReader reader = IndexReader.open(ldir);

    IndexSearcher searcher = new IndexSearcher(reader);

    for (int i = 0; i < 1000; i++) {
    Document doc = new Document();
    doc.add(new Field("uri", "http://example.org/", Field.Store.YES,
    Field.Index.NO));
    writer.addDocument(doc);
    writer.commit();

    reader.reopen();
    searcher.close();
    searcher = new IndexSearcher(reader);

    // do something here

    Term term = new Term("uri", "http://example.org/");
    writer.deleteDocuments(term);
    writer.commit();
    }

    Any idea what's going wrong?
    Doing commit operation at the end of the loop can solve your problem. It is not a good idea to commit after every single addition.




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Andreas Harth at Jan 8, 2011 at 5:39 pm
    Hi Ian, Ahmet,
    On 01/08/2011 06:13 PM, Ian Lea wrote:
    You also need to read the javadocs for reopen(), and the sample code
    there. And it would be worth reading up on lucene's near real-time
    (NRT) features.
    yep, that was it. reopen() behaves differently to what I'd expected.
    Using the IndexReader from IndexWriter.getReader() works.

    Thanks!

    Best regards,
    Andreas.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJan 8, '11 at 4:31p
activeJan 8, '11 at 5:39p
posts4
users3
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase