FAQ
Hello Group,
I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?

- RB




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Jason Rutherglen at Dec 8, 2009 at 6:31 am
    RB,

    That's expected behavior, each .cfs corresponds to all of a
    segment's files. You could write your own directory
    implementation that underneath writes to a single file. It's
    usually good to present what you're trying to accomplish (i.e.
    the why).

    Jason

    On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
    wrote:
    Hello Group,
    I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?

    - RB




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Cool The Breezer at Dec 8, 2009 at 6:43 am
    Thanks Jason for quick reply and suggestion. I am definitely going to change directory implementation policy to have one .cfs file instead of hundreds in one directory.

    regards,
    Ranjan


    ----- Original Message ----
    From: Jason Rutherglen <jason.rutherglen@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 12:00:59 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    RB,

    That's expected behavior, each .cfs corresponds to all of a
    segment's files. You could write your own directory
    implementation that underneath writes to a single file. It's
    usually good to present what you're trying to accomplish (i.e.
    the why).

    Jason

    On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
    wrote:
    Hello Group,
    I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?

    - RB




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Dec 8, 2009 at 10:01 am
    IndexWriter takes care of merging the CFSs down, over time. Have you
    changed your mergeFactor? It's odd to see 100s of CFSs.

    Or maybe you're not closing the old reader on reopening a new one?
    That would prevent deletion of the files.

    Mike

    On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
    wrote:
    Thanks Jason for quick reply and suggestion. I am definitely going to change directory implementation policy to have one .cfs file instead of hundreds in one directory.

    regards,
    Ranjan


    ----- Original Message ----
    From: Jason Rutherglen <jason.rutherglen@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 12:00:59 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    RB,

    That's expected behavior, each .cfs corresponds to all of a
    segment's files. You could write your own directory
    implementation that underneath writes to a single file. It's
    usually good to present what you're trying to accomplish (i.e.
    the why).

    Jason

    On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
    wrote:
    Hello Group,
    I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?

    - RB




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Cool The Breezer at Dec 8, 2009 at 11:14 am
    Thanks Mike for your timely suggestion. Somehow readers are not reopened properly.





    ----- Original Message ----
    From: Michael McCandless <lucene@mikemccandless.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 3:31:22 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    IndexWriter takes care of merging the CFSs down, over time. Have you
    changed your mergeFactor? It's odd to see 100s of CFSs.

    Or maybe you're not closing the old reader on reopening a new one?
    That would prevent deletion of the files.

    Mike

    On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
    wrote:
    Thanks Jason for quick reply and suggestion. I am definitely going to change directory implementation policy to have one .cfs file instead of hundreds in one directory.

    regards,
    Ranjan


    ----- Original Message ----
    From: Jason Rutherglen <jason.rutherglen@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 12:00:59 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    RB,

    That's expected behavior, each .cfs corresponds to all of a
    segment's files. You could write your own directory
    implementation that underneath writes to a single file. It's
    usually good to present what you're trying to accomplish (i.e.
    the why).

    Jason

    On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
    wrote:
    Hello Group,
    I am continuously updating an index and at the same time searcher also searches the index, which resulted in multiple .cfs files for each commit by IndexWriter. I am not sure whether this is an expected behavior or I need to merge each time after IndexWriter commits. In between the writer's commit, I reopen the IndexReader but still .cfs files keep on increasing. I would like to have one .cfs file which can be reopened frequently by reader and at the same time progressive commit by writer. Is it possible?

    - RB




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Erick Erickson at Dec 8, 2009 at 1:27 pm
    If you're using reopen, be sure to close the old reader
    if the new one isn't identical, something like:

    IndexReader new = r.reopen();
    if (new != reader) {
    ... // reader was reopened
    reader.close();
    }
    reader = new;


    Erick


    On Tue, Dec 8, 2009 at 6:13 AM, Cool The Breezer
    wrote:
    Thanks Mike for your timely suggestion. Somehow readers are not reopened
    properly.





    ----- Original Message ----
    From: Michael McCandless <lucene@mikemccandless.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 3:31:22 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    IndexWriter takes care of merging the CFSs down, over time. Have you
    changed your mergeFactor? It's odd to see 100s of CFSs.

    Or maybe you're not closing the old reader on reopening a new one?
    That would prevent deletion of the files.

    Mike

    On Tue, Dec 8, 2009 at 1:43 AM, Cool The Breezer
    wrote:
    Thanks Jason for quick reply and suggestion. I am definitely going to
    change directory implementation policy to have one .cfs file instead of
    hundreds in one directory.
    regards,
    Ranjan


    ----- Original Message ----
    From: Jason Rutherglen <jason.rutherglen@gmail.com>
    To: java-user@lucene.apache.org
    Sent: Tue, December 8, 2009 12:00:59 PM
    Subject: Re: IndexWriter creates multiple .cfs files

    RB,

    That's expected behavior, each .cfs corresponds to all of a
    segment's files. You could write your own directory
    implementation that underneath writes to a single file. It's
    usually good to present what you're trying to accomplish (i.e.
    the why).

    Jason

    On Mon, Dec 7, 2009 at 10:25 PM, Cool The Breezer
    wrote:
    Hello Group,
    I am continuously updating an index and at the same
    time searcher also searches the index, which resulted in multiple .cfs files
    for each commit by IndexWriter. I am not sure whether this is an expected
    behavior or I need to merge each time after IndexWriter commits. In between
    the writer's commit, I reopen the IndexReader but still .cfs files keep on
    increasing. I would like to have one .cfs file which can be reopened
    frequently by reader and at the same time progressive commit by writer. Is
    it possible?
    - RB




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org




    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedDec 8, '09 at 6:26a
activeDec 8, '09 at 1:27p
posts6
users4
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase