FAQ
Anyone else run on Windows? We have index around 26 GB in size. Seems file system cache ends up taking up nearly all available RAM (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB, so very little left over for queries, etc, and box starts swapping during searches. I think changing file system cache size setting is needed. Anyone else have same issue?

Search Discussions

  • Mark Miller at Aug 16, 2008 at 2:23 pm

    Robert Stewart wrote:
    Anyone else run on Windows? We have index around 26 GB in size. Seems file system cache ends up taking up nearly all available RAM (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB, so very little left over for queries, etc, and box starts swapping during searches. I think changing file system cache size setting is needed. Anyone else have same issue?
    Hmmm...get more ram :)

    Windows 64-bit upped the default file system cache size from 1 gig to 1
    terabyte. Your feeling the awesome effects of that upgrade I think.

    There is an API call ( SetSystemFilecache() ) to override this - so
    perhaps code up a C app to set it before running your Lucene app?

    - Mark

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Mark Miller at Aug 16, 2008 at 2:51 pm

    Mark Miller wrote:
    Robert Stewart wrote:
    Anyone else run on Windows? We have index around 26 GB in size.
    Seems file system cache ends up taking up nearly all available RAM
    (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
    so very little left over for queries, etc, and box starts swapping
    during searches. I think changing file system cache size setting is
    needed. Anyone else have same issue?
    Hmmm...get more ram :)

    Windows 64-bit upped the default file system cache size from 1 gig to
    1 terabyte. Your feeling the awesome effects of that upgrade I think.

    There is an API call ( SetSystemFilecache() ) to override this - so
    perhaps code up a C app to set it before running your Lucene app?

    - Mark
    You may actually be able to do it from the registry as well:
    http://support.microsoft.com/kb/892589 (don't use windows anymore so
    havn't confirmed)

    Info showing the change from 1 gig to 1 terabyte:
    http://support.microsoft.com/kb/294418

    You just want to set it to a certain percentage of what you got -
    leaving enough to do whatever your lucene app needs to do.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Mark Miller at Aug 18, 2008 at 2:04 pm

    Mark Miller wrote:
    Mark Miller wrote:
    Robert Stewart wrote:
    Anyone else run on Windows? We have index around 26 GB in size.
    Seems file system cache ends up taking up nearly all available RAM
    (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
    so very little left over for queries, etc, and box starts swapping
    during searches. I think changing file system cache size setting is
    needed. Anyone else have same issue?
    Hmmm...get more ram :)

    Windows 64-bit upped the default file system cache size from 1 gig to
    1 terabyte. Your feeling the awesome effects of that upgrade I think.

    There is an API call ( SetSystemFilecache() ) to override this - so
    perhaps code up a C app to set it before running your Lucene app?

    - Mark
    You may actually be able to do it from the registry as well:
    http://support.microsoft.com/kb/892589 (don't use windows anymore so
    havn't confirmed)

    Info showing the change from 1 gig to 1 terabyte:
    http://support.microsoft.com/kb/294418

    You just want to set it to a certain percentage of what you got -
    leaving enough to do whatever your lucene app needs to do.
    Found a great page about the problem using Domino:
    http://www-1.ibm.com/support/docview.wss?uid=swg21270452

    They appear to have compiled all the little bits of info that I have
    seen elsewhere, and describe the problem being fixed just as you did.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Robert Stewart at Aug 19, 2008 at 2:27 pm
    Thank you for the help. It seems that just changing "memory usage" setting to "programs" from default of "system cache" fixed the issue. Now it takes only about 4 GB of system cache instead of 26 GB, and search performance is back to normal (fast).

    -----Original Message-----
    From: Mark Miller
    Sent: Monday, August 18, 2008 10:03 AM
    To: java-user@lucene.apache.org
    Subject: Re: windows file system cache

    Mark Miller wrote:
    Mark Miller wrote:
    Robert Stewart wrote:
    Anyone else run on Windows? We have index around 26 GB in size.
    Seems file system cache ends up taking up nearly all available RAM
    (26 GB out of 32 GB on 64-bit box). Lucene process is around 5 GB,
    so very little left over for queries, etc, and box starts swapping
    during searches. I think changing file system cache size setting is
    needed. Anyone else have same issue?
    Hmmm...get more ram :)

    Windows 64-bit upped the default file system cache size from 1 gig to
    1 terabyte. Your feeling the awesome effects of that upgrade I think.

    There is an API call ( SetSystemFilecache() ) to override this - so
    perhaps code up a C app to set it before running your Lucene app?

    - Mark
    You may actually be able to do it from the registry as well:
    http://support.microsoft.com/kb/892589 (don't use windows anymore so
    havn't confirmed)

    Info showing the change from 1 gig to 1 terabyte:
    http://support.microsoft.com/kb/294418

    You just want to set it to a certain percentage of what you got -
    leaving enough to do whatever your lucene app needs to do.
    Found a great page about the problem using Domino:
    http://www-1.ibm.com/support/docview.wss?uid=swg21270452

    They appear to have compiled all the little bits of info that I have
    seen elsewhere, and describe the problem being fixed just as you did.

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Toke Eskildsen at Aug 18, 2008 at 7:53 am

    On Sat, 2008-08-16 at 07:40 -0400, Robert Stewart wrote:
    Anyone else run on Windows? We have index around 26 GB in size.
    Seems file system cache ends up taking up nearly all available RAM
    (26 GB out of 32 GB on 64-bit box).
    Sounds fine so far. If the RAM isn't used for anything else, the system
    should use it for caching.
    Lucene process is around 5 GB, so very little left over for queries,
    etc, and box starts swapping during searches.
    Not so fine and also unexpected. Are you sure that what you're seeing is
    swapping and not just flushing of the write-cache? Are you observing the
    disk-activity only when your switching from writer to reader or does it
    continue to be a problem with subsequent searches?

    Have you profiled the JVM memory usage? Do you have the same problem if
    you specify 6GB as the ceiling?


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Aug 18, 2008 at 9:46 am

    Toke Eskildsen wrote:

    Lucene process is around 5 GB, so very little left over for queries,
    etc, and box starts swapping during searches.
    Not so fine and also unexpected. Are you sure that what you're
    seeing is
    swapping and not just flushing of the write-cache? Are you observing
    the
    disk-activity only when your switching from writer to reader or does
    it
    continue to be a problem with subsequent searches?
    On Linux, at least, the OS will choose to evict "real" pages (swap
    them out to disk) and then use the RAM for the IO cache instead. This
    then causes lots of paging when less-often-used parts of your app need
    to access memory.

    You can tune the "swappiness" kernel parameter (http://kerneltrap.org/node/3000
    ) to change how aggressively it does this. But I've never heard of a
    corresponding way to tune Windows.

    Mike

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedAug 16, '08 at 11:44a
activeAug 19, '08 at 2:27p
posts7
users4
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase