FAQ
Hi,

Our Lucene index grew to about 4 GB .
Unfortunately it brought up a performance problem of slow file merging.
We have:
1. A writer thread: once an Hour it looks for modified documents, and
updates the Lucene index.
Usually there are only few modifications, but sometimes we switch the
entire content and re-index everything.

2. The default Lucene Merge thread (ConcurrentMergeScheduler)

Usually it works great. But every several hours the
'ConcurrentMergeScheduler' thread gets stuck (for hours - I'm guessing
it got to the point where it needs to merge large files).
During this, our Writer thread is stuck (waiting on a lock), so users
will see stale data.

My questions please:

1. Is there any configuration that would either speed up file merging,
or allow IndexWriter to write simultaneously?

2. And when do I call 'optimize'?
Won't it be another very operation, that holds the 'write' lock and
prevents updates?

Thanks:)

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Ian Lea at Jun 15, 2011 at 3:45 pm
    Waits of several hours on a 4Gb index sounds very unlikely. Are you
    sure there isn't something else going on that is blocking things?
    What version of lucene? Decent, error-free, hardware?

    As for optimize, I'd skip it altogether, or schedule it occasionally
    when there is no or low activity on the index.


    --
    Ian.

    On Wed, Jun 15, 2011 at 1:59 PM, sol myr wrote:
    Hi,

    Our Lucene index grew to about 4 GB .
    Unfortunately it brought up a performance problem of slow file merging.
    We have:
    1. A writer thread: once an Hour it looks for modified documents, and
    updates the Lucene index.
    Usually there are only few modifications, but sometimes we switch the
    entire content and re-index everything.

    2. The default Lucene Merge thread (ConcurrentMergeScheduler)

    Usually it works great. But every several hours the
    'ConcurrentMergeScheduler' thread gets stuck (for hours - I'm guessing
    it got to the point where it needs to merge large files).
    During this, our Writer thread is stuck (waiting on a lock), so users
    will see stale data.

    My questions please:

    1. Is there any configuration that would either speed up file merging,
    or allow IndexWriter to write simultaneously?

    2. And when do I call 'optimize'?
    Won't it be another very operation, that holds the 'write' lock and
    prevents updates?

    Thanks:)

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJun 15, '11 at 1:00p
activeJun 15, '11 at 3:45p
posts2
users2
websitelucene.apache.org

2 users in discussion

Ian Lea: 1 post Sol myr: 1 post

People

Translate

site design / logo © 2022 Grokbase