FAQ
Hi,
other than the in memory terms (.tii), and the few kilobytes of opened file buffer, where are some other sources of significant memory consumption
when searching on a large index ? (> 100GB). The queries are just normal term queries.


_________________________________________________________________
隨身的 Windows Live Messenger 和 Hotmail,不限時地掌握資訊盡在指間 — Windows Live for Mobile
http://www.msn.com.tw/msnmobile/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Jian chen at May 29, 2008 at 10:26 pm
    Not that I can think about. But, if you have any cached field data,
    norms array, that could be huge.

    Would be interested in knowing from others regarding this topic as well.

    Jian
    On 5/29/08, Alex wrote:

    Hi,
    other than the in memory terms (.tii), and the few kilobytes of opened file
    buffer, where are some other sources of significant memory consumption
    when searching on a large index ? (> 100GB). The queries are just normal
    term queries.


    _________________________________________________________________
    隨身的 Windows Live Messenger 和 Hotmail,不限時地掌握資訊盡在指間 — Windows Live for Mobile
    http://www.msn.com.tw/msnmobile/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Alex at May 29, 2008 at 10:31 pm
    Currently, searching on our index consumes around 2.5GB of ram.
    This is just a single term query, nothing that requires the in memory cache like in
    the FieldScoreQuery.


    Alex

    ----------------------------------------
    Date: Thu, 29 May 2008 15:25:43 -0700
    From: chenjian1227@gmail.com
    To: java-user@lucene.apache.org
    Subject: Re: lucene memory consumption

    Not that I can think about. But, if you have any cached field data,
    norms array, that could be huge.

    Would be interested in knowing from others regarding this topic as well.

    Jian
    On 5/29/08, Alex wrote:

    Hi,
    other than the in memory terms (.tii), and the few kilobytes of opened file
    buffer, where are some other sources of significant memory consumption
    when searching on a large index ? (> 100GB). The queries are just normal
    term queries.


    _________________________________________________________________
    隨身的 Windows Live Messenger 和 Hotmail,不限時地掌握資訊盡在指間 — Windows Live for Mobile
    http://www.msn.com.tw/msnmobile/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    _________________________________________________________________
    隨身的 Windows Live Messenger 和 Hotmail,不限時地掌握資訊盡在指間 — Windows Live for Mobile
    http://www.msn.com.tw/msnmobile/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Mark Miller at May 29, 2008 at 10:40 pm

    Alex wrote:
    Currently, searching on our index consumes around 2.5GB of ram.
    This is just a single term query, nothing that requires the in memory cache like in
    the FieldScoreQuery.


    Alex

    ----------------------------------------
    That seems rather high. You have 10/15 million + docs?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Alex at May 29, 2008 at 11:06 pm
    I believe we have around 346 million documents


    Alex

    ----------------------------------------
    Date: Thu, 29 May 2008 18:39:31 -0400
    From: markrmiller@gmail.com
    To: java-user@lucene.apache.org
    Subject: Re: lucene memory consumption

    Alex wrote:
    Currently, searching on our index consumes around 2.5GB of ram.
    This is just a single term query, nothing that requires the in memory cache like in
    the FieldScoreQuery.


    Alex

    ----------------------------------------
    That seems rather high. You have 10/15 million + docs?

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    _________________________________________________________________
    聰明搜尋和瀏覽網路的免費工具列 — MSN 搜尋工具列
    http://toolbar.live.com/

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Yonik Seeley at May 29, 2008 at 11:16 pm

    2008/5/29 Alex <chy1013m1@hotmail.com>:
    I believe we have around 346 million documents
    So that would be 346MB per indexed field that you search.

    Also, if you sort on anything other than score, that will take up a
    lot of memory to un-invert the field.

    -Yonik

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Daniel Noll at May 29, 2008 at 10:57 pm

    On Friday 30 May 2008 08:17:52 Alex wrote:
    Hi,
    other than the in memory terms (.tii), and the few kilobytes of opened file
    buffer, where are some other sources of significant memory consumption when
    searching on a large index ? (> 100GB). The queries are just normal term
    queries.
    Norms can also eat quite a bit of space if you have a large number of fields.

    Daniel

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedMay 29, '08 at 10:18p
activeMay 29, '08 at 11:16p
posts7
users5
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase