FAQ
Hi,

I am using Lucene 2.9.4 with FSDirectory.
My index has 80 thousand documents (each document has 12 fields).
My jvm has 70Mb of RAM memory (limited by my hosting).
I am getting various OutOfMemoryError.
I ran jmap and I got:

num #instances #bytes Class description
--------------------------------------------------------------------------
1: 275926 32074080 char[]
2: 657724 31570752
org.apache.lucene.index.FreqProxTermsWriter$PostingList
3: 200 16094896 org.apache.lucene.index.RawPostingList[]

The Lucene is consuming 47Mb of RAM memory.
Why there is this excessive memory consumption in FSDirectory?
Is it correct? I'm using FSDirectory because I haven´t a big amount of
RAM memory.

Thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Erick Erickson at Apr 4, 2011 at 11:38 am
    FSDirectory will, indeed, store the index on disk. However,
    when *using* that index, lots of stuff happens. Specifically:
    When indexing, there is a buffer that accumulates documents
    until it's flushed to disk. Are you indexing?

    When searching (and this is the more important part), various
    caches are used to speed up searches. A substantial part of
    your index may be held in memory at various points.

    70M just isn't very much memory, I think you'll have to get
    more if at all possible.

    Best
    Erick
    On Mon, Apr 4, 2011 at 7:13 AM, Claudio wrote:

    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num #instances #bytes Class description
    --------------------------------------------------------------------------
    1: 275926 32074080 char[]
    2: 657724 31570752
    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3: 200 16094896 org.apache.lucene.index.RawPostingList[]

    The Lucene is consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Is it correct? I'm using FSDirectory because I haven´t a big amount of RAM
    memory.

    Thanks.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Claudio at Apr 5, 2011 at 5:18 am
    Ok Erick,

    Thanks for your quick answer.
    FSDirectory will, indeed, store the index on disk. However,
    when *using* that index, lots of stuff happens. Specifically:
    When indexing, there is a buffer that accumulates documents
    until it's flushed to disk. Are you indexing?

    When searching (and this is the more important part), various
    caches are used to speed up searches. A substantial part of
    your index may be held in memory at various points.

    70M just isn't very much memory, I think you'll have to get
    more if at all possible.

    Best
    Erick
    On Mon, Apr 4, 2011 at 7:13 AM, Claudio wrote:

    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num #instances #bytes Class description
    --------------------------------------------------------------------------
    1: 275926 32074080 char[]
    2: 657724 31570752
    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3: 200 16094896 org.apache.lucene.index.RawPostingList[]

    The Lucene is consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Is it correct? I'm using FSDirectory because I haven´t a big amount of RAM
    memory.

    Thanks.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Claudio R at Apr 5, 2011 at 4:44 pm
    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num       #instances    #bytes    Class description
    --------------------------------------------------------------------------
    1:        275926    32074080    char[]
    2:        657724    31570752    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3:        200    16094896    org.apache.lucene.index.RawPostingList[]

    The Lucene is consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Thanks.
  • Claudio R at Apr 5, 2011 at 4:45 pm
    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num       #instances    #bytes    Class description
    --------------------------------------------------------------------------
    1:        275926    32074080    char[]
    2:        657724    31570752    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3:        200    16094896    org.apache.lucene.index.RawPostingList[]

    The Lucene is
    consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Thanks.
  • Claudio R at Apr 5, 2011 at 4:45 pm
    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num       #instances    #bytes    Class description
    --------------------------------------------------------------------------
    1:        275926    32074080    char[]
    2:        657724    31570752    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3:        200    16094896    org.apache.lucene.index.RawPostingList[]

    The Lucene is
    consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Is it correct?
    Thanks.
  • Michael McCandless at Apr 5, 2011 at 5:04 pm
    Try 1) reducing the RAM buffer of your IndexWriter
    (IndexWriter.setRAMBufferSizeMB), 2) using a term divisor when opening
    your reader (pass 2 or 3 or 4 as termInfosIndexDivisor when opening
    IndexReader), and 3) disabling norms or not indexing as many fields as
    possible.

    70Mb is not that much RAM to work with, but, for only 80K docs it
    should be possible as long as the docs are not large.

    Mike

    http://blog.mikemccandless.com
    On Mon, Apr 4, 2011 at 6:39 AM, Claudio R wrote:
    Hi,

    I am using Lucene 2.9.4 with FSDirectory.
    My index has 80 thousand documents (each document has 12 fields).
    My jvm has 70Mb of RAM memory (limited by my hosting).
    I am getting various OutOfMemoryError.
    I ran jmap and I got:

    num       #instances    #bytes    Class description
    --------------------------------------------------------------------------
    1:        275926    32074080    char[]
    2:        657724    31570752    org.apache.lucene.index.FreqProxTermsWriter$PostingList
    3:        200    16094896    org.apache.lucene.index.RawPostingList[]

    The Lucene is
    consuming 47Mb of RAM memory.
    Why there is this excessive memory consumption in FSDirectory?
    Is it correct?
    Thanks.
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedApr 4, '11 at 11:13a
activeApr 5, '11 at 5:04p
posts7
users4
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase