FAQ
I've got an application that performs millions of searches against a
lucene index, can someone give me a bit of insight as to the memory
consumption of these searches? Is there a cap on how many are kept
around? is there any way I can disable caching for this type of
search?

--
___________________________________________________
Chris Fraschetti
e fraschetti@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Otis Gospodnetic at Aug 3, 2005 at 8:04 pm
    Searches are not cached (other than possibly by the OS/File system).
    IndexSearcher can/should be cached (by your application), and how much
    memory it consumes depends on ... hm, I thought we included a formula
    for that in Lucene in Action, but it looks like we only included the
    formula for calculating the number of open files during indexing.

    Well, Doug provided this formula a few times, and I don't recall it off
    the top of my head, but I do know that the .tii index file will be
    fully read into memory, so look at the size of that file to get an
    initial idea.

    Otis

    --- Chris Fraschetti wrote:
    I've got an application that performs millions of searches against a
    lucene index, can someone give me a bit of insight as to the memory
    consumption of these searches? Is there a cap on how many are kept
    around? is there any way I can disable caching for this type of
    search?

    --
    ___________________________________________________
    Chris Fraschetti
    e fraschetti@gmail.com

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Chris Hostetter at Aug 3, 2005 at 11:04 pm
    : Well, Doug provided this formula a few times, and I don't recall it off
    : the top of my head, but I do know that the .tii index file will be
    : fully read into memory, so look at the size of that file to get an
    : initial idea.

    the other big piece of caching that goes on is the FieldCache which is
    used for sorting -- if you only ever sort by Score or doc ID, then it's a
    non-issue, but if your millions of Searchers are going to involve sorting
    on lots of different fields, the complete list of indexed values for each
    field used in sorting will be cached in memory per Searcher (close the
    Searcher, the WeakRef for the FieldCache is freed up)

    I can imagine some scenerios in which it might make sense to do all of the
    searches that sort on field A first, then close the searcher and open a
    new one and do all the searchers that sort on field B, then close the
    searcher, open a new one and do all the searches that sort on field C,
    etc....


    : > I've got an application that performs millions of searches against a
    : > lucene index, can someone give me a bit of insight as to the memory
    : > consumption of these searches? Is there a cap on how many are kept
    : > around? is there any way I can disable caching for this type of
    : > search?
    : >
    : > --
    : > ___________________________________________________
    : > Chris Fraschetti
    : > e fraschetti@gmail.com
    : >
    : > ---------------------------------------------------------------------
    : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    : > For additional commands, e-mail: java-user-help@lucene.apache.org
    : >
    : >
    :
    :
    : ---------------------------------------------------------------------
    : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    : For additional commands, e-mail: java-user-help@lucene.apache.org
    :



    -Hoss


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • David Spencer at Aug 3, 2005 at 11:51 pm

    Chris Fraschetti wrote:

    I've got an application that performs millions of searches against a
    If the results are not, say, "personalized", than I suggest some kind of
    web container cache - I use and like OSCache - and it can even cache
    page fragments.

    http://www.opensymphony.com/oscache/
    lucene index, can someone give me a bit of insight as to the memory
    consumption of these searches? Is there a cap on how many are kept
    around? is there any way I can disable caching for this type of
    search?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedAug 3, '05 at 7:26p
activeAug 3, '05 at 11:51p
posts4
users4
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase