FAQ
Hi There

I have an index which is 36 GB large. When I perform eight simultaneous
searches (performed by JMeter) on the index, an OutOfMemory error
occurs. Since I need to potentially search across multiple indexes and
those indexes can change from one search query to the next, each user
has their own ParallelMultiSearcher object. Before each search
operation, I reconstruct the ParrallelMultisearcher with the appropriate
Searchers to each of the indexes that need to be included for that
particular search query.

The problem is that requiring each user to have their own
ParallelMultisearcher seems to limit the number of users that can use
the system at the same time.

While experimenting, when I make the ParallelMultiSearcher static, the
same object used by all users, the OutOfMemory problem goes away and I
am able to execute 50 simultaneous searches. The problem I have is I
cannot make ParallelMultisearcher static, since the specific indexes
used are variable from one search query to the next. I initially thought
one could just cache the underlying Searchers and all would be okay, but
this does not appear to be the case.

My question: Will ParallelMultisearcher tend to consume a large amount
of memory by itself when used on large indices? If so, do you have any
suggestions on how I might support the above scenario (i.e. when the
indexes used change from one query to the next)

Thanks in advance

Jamie


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Ian Lea at Mar 17, 2010 at 11:21 am
    Hi


    Caching searchers at some level should help keep memory usage down -
    and will help performance too. Searchers themselves don't generally
    consume large amounts of memory, but if you've got loads of them then
    obviously things will add up.

    Unless you can change the whole design of your app (single index with
    a user field that you use as a query filter to restrict users to
    "their" data?) you may be stuck with giving the app more memory or
    restricting the number of concurrent searchers.


    ---
    Ian.

    On Tue, Mar 16, 2010 at 10:00 PM, Jamie wrote:
    Hi There

    I have an index which is 36 GB large. When I perform  eight simultaneous
    searches (performed by JMeter) on the index, an OutOfMemory error occurs.
    Since I need to potentially search across multiple indexes and those indexes
    can change from one search query to the next, each user has their own
    ParallelMultiSearcher object. Before each search operation, I reconstruct
    the ParrallelMultisearcher with the appropriate Searchers to each of the
    indexes that need to be included for that particular search query.

    The problem is that requiring each user to have their own
    ParallelMultisearcher seems to limit the number of  users that can use the
    system at the same time.

    While experimenting, when I make the ParallelMultiSearcher static, the same
    object used by all users, the OutOfMemory problem goes away and I am able to
    execute 50 simultaneous searches. The problem I have is I cannot make
    ParallelMultisearcher static, since the specific indexes used are variable
    from one search query to the next. I initially thought one could just cache
    the underlying Searchers and all would be okay, but this does not appear to
    be the case.

    My question: Will ParallelMultisearcher tend to consume a large amount of
    memory by itself when used on large indices? If so, do you have any
    suggestions on how I might support the above scenario (i.e. when the indexes
    used change from one query to the next)

    Thanks in advance

    Jamie


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Jamie at Mar 18, 2010 at 6:01 am
    Hi Ian

    Thanks for the info. Its difficult to reuse searchers as my users are
    performing realtime searches, so I need to open an IndexReader for every
    live search query.

    I've since tracked the OutOfMemory issue down to sort on date. I am
    using too high a precision (down to the second) which is causing the
    number of search terms to escalate. Also, I haven't since moved the date
    field in the index over to using Numerics instead of a String value for
    fear of breaking compatibility with my old index format.

    I dont yet quite understand the implications of the precisionStep and
    what it all means. If I change my date string to a Numeric integer in
    the format yyyyMMddHHmm, what should the precisionStep value be?

    Jamie


    On 2010/03/17 01:20 PM, Ian Lea wrote:
    Hi


    Caching searchers at some level should help keep memory usage down -
    and will help performance too. Searchers themselves don't generally
    consume large amounts of memory, but if you've got loads of them then
    obviously things will add up.

    Unless you can change the whole design of your app (single index with
    a user field that you use as a query filter to restrict users to
    "their" data?) you may be stuck with giving the app more memory or
    restricting the number of concurrent searchers.


    ---
    Ian.


    On Tue, Mar 16, 2010 at 10:00 PM, Jamiewrote:
    Hi There

    I have an index which is 36 GB large. When I perform eight simultaneous
    searches (performed by JMeter) on the index, an OutOfMemory error occurs.
    Since I need to potentially search across multiple indexes and those indexes
    can change from one search query to the next, each user has their own
    ParallelMultiSearcher object. Before each search operation, I reconstruct
    the ParrallelMultisearcher with the appropriate Searchers to each of the
    indexes that need to be included for that particular search query.

    The problem is that requiring each user to have their own
    ParallelMultisearcher seems to limit the number of users that can use the
    system at the same time.

    While experimenting, when I make the ParallelMultiSearcher static, the same
    object used by all users, the OutOfMemory problem goes away and I am able to
    execute 50 simultaneous searches. The problem I have is I cannot make
    ParallelMultisearcher static, since the specific indexes used are variable
    from one search query to the next. I initially thought one could just cache
    the underlying Searchers and all would be okay, but this does not appear to
    be the case.

    My question: Will ParallelMultisearcher tend to consume a large amount of
    memory by itself when used on large indices? If so, do you have any
    suggestions on how I might support the above scenario (i.e. when the indexes
    used change from one query to the next)

    Thanks in advance

    Jamie


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedMar 16, '10 at 10:00p
activeMar 18, '10 at 6:01a
posts3
users2
websitelucene.apache.org

2 users in discussion

Jamie: 2 posts Ian Lea: 1 post

People

Translate

site design / logo © 2022 Grokbase