FAQ
Hi all,

We are using Lucene 2.9.0 to power a search and are performing a large
number of searches. We are initializing our searcher with the code:

Directory directory = FSDirectory.open(new File(Constants.IndexLocation));
Searcher searcher = new IndexSearcher(directory,true);

Upon every search request, we are performing the following

TopDocs results = searcher.search(query,Constants.MAXHITS);

As expected, the first request takes up a large amount of time as the index
warms up. However, we have noticed that the time taken keeps decreasing
continuously as and when more results are served. My guess is that results
are being cached somewhere.

The runtime classes for directory and index reader are as follows:
org.apache.lucene.store.NIOFSDirectory
org.apache.lucene.index.ReadOnlyDirectoryReader

Eventually, JVM runs out of memory as we see from jvm stats that old
generation capacity reaches the maximum. We are using Concurrent Mark sweep
GC strategy, and it gets invoked during every search request thereafter.

I have tried looking for answers in forums, but couldn't find any. As per
many threads, there is no caching involved in Lucene by default, unless
QueryFilter and CachingWrapperFilter are used. I tried looking into Lucene
code also a bit, but couldn't find any clues.

Does anyone have any pointers as to why this could be happening ?

Thanks!
Abhay

--
View this message in context: http://lucene.472066.n3.nabble.com/Memory-leak-with-Lucene-Search-tp3162070p3162070.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Ian Lea at Jul 12, 2011 at 11:25 am
    Lucene does not cache results. Operating systems do cache things and
    on unix anyway (no idea about windows) some speedups over time can
    reasonably be attributed to disk caching by the OS.

    Have you profiled your app to find out exactly what is using the
    memory? Do you just use the one searcher or are you opening new ones
    from time to time? If the latter, are you closing the old ones?


    --
    Ian.

    On Tue, Jul 12, 2011 at 11:52 AM, dangabhay wrote:
    Hi all,

    We are using Lucene 2.9.0 to power a search and are performing a large
    number of searches. We are initializing our searcher with the code:

    Directory directory = FSDirectory.open(new File(Constants.IndexLocation));
    Searcher searcher = new IndexSearcher(directory,true);

    Upon every search request, we are performing the following

    TopDocs results = searcher.search(query,Constants.MAXHITS);

    As expected, the first request takes up a large amount of time as the index
    warms up. However, we have noticed that the time taken keeps decreasing
    continuously as and when more results are served. My guess is that results
    are being cached somewhere.

    The runtime classes for directory and index reader are as follows:
    org.apache.lucene.store.NIOFSDirectory
    org.apache.lucene.index.ReadOnlyDirectoryReader

    Eventually, JVM runs out of memory as we see from jvm stats that old
    generation capacity reaches the maximum. We are using Concurrent Mark sweep
    GC strategy, and it gets invoked during every search request thereafter.

    I have tried looking for answers in forums, but couldn't find any. As per
    many threads, there is no caching involved in Lucene by default, unless
    QueryFilter and CachingWrapperFilter are used. I tried looking into Lucene
    code also a bit, but couldn't find any clues.

    Does anyone have any pointers as to why this could be happening ?

    Thanks!
    Abhay

    --
    View this message in context: http://lucene.472066.n3.nabble.com/Memory-leak-with-Lucene-Search-tp3162070p3162070.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJul 12, '11 at 10:52a
activeJul 12, '11 at 11:25a
posts2
users2
websitelucene.apache.org

2 users in discussion

Ian Lea: 1 post Dangabhay: 1 post

People

Translate

site design / logo © 2022 Grokbase