FAQ
Hi all,

I have a problem that my index files is 33.4gb now, and when I insert new
index through Lucene.Net few times. Lucene will execute merge index files
automatically. The problem is during merge executed, writer.lock file will
be created and all operations such as read/write index files were all
stopped.
(for a long time and there were no any status or flag to know Lucene were
executeing merge process.)

Are there any solutions to avoid this "long time lock" situation?

Thanks in advanced.

Floyd

Search Discussions

  • Michael Garski at Mar 8, 2010 at 4:20 pm
    Floyd,

    Long-running merges are an issue with large indexes due to the CPU and
    I/O overhead required to perform the merge, but there are a few things
    you can do to reduce that time:

    1) Create a custom merge policy to give you control over when merges
    occur and what segments are merged
    2) Shard your index into smaller pieces that are searched together using
    a MultiSearcher - this will keep your indexes smaller.

    Michael

    -----Original Message-----
    From: Floyd Wu
    Sent: Monday, March 08, 2010 1:45 AM
    To: lucene-net-user@incubator.apache.org
    Subject: About processing handling during merge time

    Hi all,

    I have a problem that my index files is 33.4gb now, and when I insert
    new
    index through Lucene.Net few times. Lucene will execute merge index
    files
    automatically. The problem is during merge executed, writer.lock file
    will
    be created and all operations such as read/write index files were all
    stopped.
    (for a long time and there were no any status or flag to know Lucene
    were
    executeing merge process.)

    Are there any solutions to avoid this "long time lock" situation?

    Thanks in advanced.

    Floyd
  • Floyd Wu at Mar 10, 2010 at 3:15 am
    Hi Michael,

    If I want to create a custom merge policy that act "Never do merge during
    work day". Where to start? I read something about merge policy could be
    related with scheduler, but I'm not sure the relationship between these two
    objects.

    many thanks.

    Floyd



    2010/3/9 Michael Garski <mgarski@myspace-inc.com>
    Floyd,

    Long-running merges are an issue with large indexes due to the CPU and
    I/O overhead required to perform the merge, but there are a few things
    you can do to reduce that time:

    1) Create a custom merge policy to give you control over when merges
    occur and what segments are merged
    2) Shard your index into smaller pieces that are searched together using
    a MultiSearcher - this will keep your indexes smaller.

    Michael

    -----Original Message-----
    From: Floyd Wu
    Sent: Monday, March 08, 2010 1:45 AM
    To: lucene-net-user@incubator.apache.org
    Subject: About processing handling during merge time

    Hi all,

    I have a problem that my index files is 33.4gb now, and when I insert
    new
    index through Lucene.Net few times. Lucene will execute merge index
    files
    automatically. The problem is during merge executed, writer.lock file
    will
    be created and all operations such as read/write index files were all
    stopped.
    (for a long time and there were no any status or flag to know Lucene
    were
    executeing merge process.)

    Are there any solutions to avoid this "long time lock" situation?

    Thanks in advanced.

    Floyd
  • Michael Garski at Mar 10, 2010 at 6:15 pm
    Floyd,

    Implementing a custom merge policy involves a concrete implementation
    Lucene.Net.Index.MergePolicy. The only documentation to point you to
    are the javadocs as the API between Java Lucene & Lucene.Net are
    identical:

    http://lucene.apache.org/java/2_9_2/api/all/org/apache/lucene/index/Merg
    ePolicy.html

    For an implementation example take a look at
    Lucene.Net.Index.LogMergePolicy.

    Michael

    -----Original Message-----
    From: Floyd Wu
    Sent: Tuesday, March 09, 2010 7:14 PM
    To: lucene-net-user@lucene.apache.org
    Subject: Re: About processing handling during merge time

    Hi Michael,

    If I want to create a custom merge policy that act "Never do merge
    during
    work day". Where to start? I read something about merge policy could be
    related with scheduler, but I'm not sure the relationship between these
    two
    objects.

    many thanks.

    Floyd



    2010/3/9 Michael Garski <mgarski@myspace-inc.com>
    Floyd,

    Long-running merges are an issue with large indexes due to the CPU and
    I/O overhead required to perform the merge, but there are a few things
    you can do to reduce that time:

    1) Create a custom merge policy to give you control over when merges
    occur and what segments are merged
    2) Shard your index into smaller pieces that are searched together using
    a MultiSearcher - this will keep your indexes smaller.

    Michael

    -----Original Message-----
    From: Floyd Wu
    Sent: Monday, March 08, 2010 1:45 AM
    To: lucene-net-user@incubator.apache.org
    Subject: About processing handling during merge time

    Hi all,

    I have a problem that my index files is 33.4gb now, and when I insert
    new
    index through Lucene.Net few times. Lucene will execute merge index
    files
    automatically. The problem is during merge executed, writer.lock file
    will
    be created and all operations such as read/write index files were all
    stopped.
    (for a long time and there were no any status or flag to know Lucene
    were
    executeing merge process.)

    Are there any solutions to avoid this "long time lock" situation?

    Thanks in advanced.

    Floyd

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouplucene-net-user @
categorieslucene
postedMar 8, '10 at 9:45a
activeMar 10, '10 at 6:15p
posts4
users2
websitelucene.apache.org

2 users in discussion

Michael Garski: 2 posts Floyd Wu: 2 posts

People

Translate

site design / logo © 2022 Grokbase