FAQ
[ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714948#action_12714948 ]

Michael McCandless commented on LUCENE-1313:
--------------------------------------------

{quote}
I want to run the Lucene unit tests in NRT mode without creating and/or
modifying all the test cases.
{quote}
You can just temporarily set the default then run all tests?

We never reached consensus on a back compat sort of setting...

{quote}
I ran a test case successfully that writes to a file while
opening threads that read from flushed sections on windows.
{quote}

Excellent; let's take this up under a new issue? If this holds true across platform (and Windows is the most challenging one) then it'll make sharing much easier.

{quote}
Closing docstores for every flush would seem to cause a lot of
overhead. With NRT + FSD aren't termvector files merged on disk for
every segment?
{quote}

Yes it adds overhead, though, it's not on the critical path for opening a new reader since it happens in the BG. So... things like LUCENE-1526 (making deletions, norms incrementally copy-on-write) I think are higher priority.

bq. Doesn't FSDir open only one FD per file?

No, it opens a real file every time openInput is called. I guess we could think about having it share/clone internally?
Realtime Search
---------------

Key: LUCENE-1313
URL: https://issues.apache.org/jira/browse/LUCENE-1313
Project: Lucene - Java
Issue Type: New Feature
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Priority: Minor
Fix For: 2.9

Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


Enable near realtime search in Lucene without external
dependencies. When RAM NRT is enabled, the implementation adds a
RAMDirectory to IndexWriter. Flushes go to the ramdir unless
there is no available space. Merges are completed in the ram
dir until there is no more available ram.
IW.optimize and IW.commit flush the ramdir to the primary
directory, all other operations try to keep segments in ram
until there is no more space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Search Discussions

  • Jason Rutherglen (JIRA) at Jun 1, 2009 at 4:26 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714966#action_12714966 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    {quote}You can just temporarily set the default then run all
    tests? {quote}

    Or have separate parallel classes inherited classes that set the
    static IW variable, run the tests, and perhaps have additional
    NRT specific test methods.

    {quote}I guess we could think about having it share/clone
    internally?{quote}

    Yeah we should, I got this confused with how we're cloning in
    most o.a.l.i classes, but if we're calling openInput it seems
    like we should have it internally clone, I guess we'll have an
    issue with classes that don't IndexInput.close, however it's
    better to solve these than open many unnecessary file
    descriptors.

    {quote}let's take this up under a new issue?{quote}

    The issue would only be the unit test for now, or should it be a
    part of an existing issue? Ok, it will clean up LUCENE-1313's
    unit test class.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 1, 2009 at 7:55 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715221#action_12715221 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    With flushToRAM=true, when IW.commit is called, do we still want
    to not have concurrent merges execute synchronously? Or only
    wait for the merges to complete that are from ramDir to
    primaryDir?
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 1, 2009 at 11:53 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715305#action_12715305 ]

    Michael McCandless commented on LUCENE-1313:
    --------------------------------------------

    {quote}
    The issue would only be the unit test for now, or should it be a
    part of an existing issue?
    {quote}
    I meant the issue of FSDir sharing a single IndexInput if the same file is opened more than once (you opened LUCENE-1671 for this).

    bq. I guess we'll have an issue with classes that don't IndexInput.close

    This (failing to close an IndexInput that was obtained from Directory.openInput) should be rare in Lucene -- we try not to do that. Failing to close clones does happen...

    {quote}
    With flushToRAM=true, when IW.commit is called, do we still want
    to not have concurrent merges execute synchronously? Or only
    wait for the merges to complete that are from ramDir to
    primaryDir?
    {quote}

    I think only ramDir -> primaryDir? commit() today doens't block on BG merges.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 2, 2009 at 7:01 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715628#action_12715628 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    {quote}For IW.resolveRAMSegments, maybe we should make a single
    merge that merges everything down? Why even bother interacting
    with a merge policy, here?{quote}

    LogMergePolicy.findMergesForOptimize I assumed would merge
    segments to a single segment, in testing resolveRAMSegments that
    doesn't always seem to be the case?

    When I created a mergeAllSegments to one, the
    ensureContiguousMerge would throw an exception. I thought about
    avoiding ensureContiguousMerge but it required reworking a bunch
    of code. Why does ensureContiguousMerge exist? Is it for
    assertions or is there a reason segments need to be next to each
    other when merging?
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 2, 2009 at 7:31 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715635#action_12715635 ]

    Michael McCandless commented on LUCENE-1313:
    --------------------------------------------

    {quote}
    LogMergePolicy.findMergesForOptimize I assumed would merge
    segments to a single segment, in testing resolveRAMSegments that
    doesn't always seem to be the case?
    {quote}
    It should, but it will respect mergeFactor in the process (ie do multiple merges if necessary). I'm thinking we should just merge all the RAM segments in one go, and not consult merge policy, in resolveRAMSegments.

    {quote}
    When I created a mergeAllSegments to one, the
    ensureContiguousMerge would throw an exception.
    {quote}
    Why does this throw an exception? Ie you passed in the in-order RAM segments?

    {quote}
    Why does ensureContiguousMerge exist? Is it for
    assertions or is there a reason segments need to be next to each
    other when merging?
    {quote}

    I think the only thing that'd break is IndexWriter now assumes continuity when it removes the old segments and puts the new one in. LUCENE-1076 explores allowing MergePolicy to select non-contiguous merges.

    But: with autoCommit=false, in order to avoid merging the doc stores, the segments (even RAM segments) must be contiguous. This is a sizable performance gain when building a large index in one IndexWriter session.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 2, 2009 at 8:45 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715675#action_12715675 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    Just so we don't forget, we need to test flushToRAM with a custom IndexDeletionPolicy which could be a bit tricky.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 5, 2009 at 4:45 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    * RAM buffer size is stored in the writer rather than set into
    DocumentsWriter. This is due to the actual ram buffer limit in
    NRT changing depending on the size of the ramdir.

    * NRTMergePolicy and IW.resolveRAMSegments merges all ram dir
    segments to primaryDir (i.e. disk) when the ramDir is over
    totalMax, or any new merges would put ramDir over totalMax.

    * In DocumentsWriter we have a set limit on the buffer size
    which is (tempMax - ramDirSize)/2. This keeps the total ram used
    under the totalMax (or IW.maxBufferSize), while also keeping our
    temporary ram usage under the tempMax amount. When DW.ramBuffer
    limit is reached, it's auto flushed to the ramDir.

    * All tests pass except TestIndexWriterRAMDir.testFSDirectory.
    Will look into this further. When flushToRAM is on by default,
    there seems to be deadlock in
    org.apache.lucene.TestMergeSchedulerExternal, however when I
    tried to see if there is any via jconsole by setting
    ANT_OPTS="-Dcom.sun.management.jmxremote" I didn't see any. I'm
    not sure if this is due to not connecting to the right process?
    Or something else.

    * Added testReadDocuments which insures we can read documents
    we've flushed to disk. This essentially tests our ability to
    simultaneously read and write documents to and from the
    docstore. It seemd to work on Windows.

    * I think there's more that can be done to more accurately
    manage the RAM however I think the way it works is a good
    starting point.


    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Mark Miller (JIRA) at Jun 15, 2009 at 1:58 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719561#action_12719561 ]

    Mark Miller commented on LUCENE-1313:
    -------------------------------------

    Whats the verdict on this one Mike? Got the impression this was a likely 3.1 ...
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 2.9

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 15, 2009 at 3:36 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Michael McCandless updated LUCENE-1313:
    ---------------------------------------

    Fix Version/s: (was: 2.9)
    3.1

    OK let's push it to 3.1. It's very much in progress, but 1) the iterations are slow (it's a big patch), 2) it's a biggish change so I'd prefer to it shortly after a release, not shortly before, so it has plenty of time to "bake" on trunk.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 15, 2009 at 5:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Summary: Near Realtime Search (was: Realtime Search)
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 15, 2009 at 5:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719665#action_12719665 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    Just wanted to give an update on this, I'm running the unit
    tests with flushToRAM=true, the ones that fail are (mostly)
    tests that look for files when they're now in RAM (temporarily)
    and the like. I'm not sure what to do with these tests, 1)
    ignore them (kind of hard to not run specific methods, I think)
    2) or conditionalize them to run only if flushToRAM=false.
    Realtime Search
    ---------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 15, 2009 at 8:58 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719767#action_12719767 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    TestThreadedOptimize is throwing a ensureContiguousMerge
    exception. I think this is highlighting the change to merging
    all ram segments to a single primaryDir segment can sometimes
    lead to choosing segments that are non-contiguous? I'm not sure
    of the best way to handle this.
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 16, 2009 at 2:27 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720147#action_12720147 ]

    Michael McCandless commented on LUCENE-1313:
    --------------------------------------------

    {quote}
    I think this is highlighting the change to merging
    all ram segments to a single primaryDir segment can sometimes
    lead to choosing segments that are non-contiguous?
    {quote}
    I don't see why that results in a non-contiguous merge? Ie, at all times the RAM segments should be on the tail of SegmentInfos? So if you merge all of them, in order, that merge should be contiguous?
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 16, 2009 at 2:27 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720148#action_12720148 ]

    Michael McCandless commented on LUCENE-1313:
    --------------------------------------------

    bq. conditionalize them to run only if flushToRAM=false.

    That seems good?
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 18, 2009 at 12:56 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    The patch is cleaned up. A static variable IndexWriter.GLOBALNRT
    is added, which allows all the tests to be run with
    flushToRAM=true. I reran the tests which hopefully still work as
    intended. Tests that looked for specific file names were changed
    to work with NRT. Some of the tests are skipped entirely and
    need to be written specifically for flushToRAM.

    * TestIndexWriterMergePolicy,TestBackwardsCompatibility failures
    are expected

    * TestIndexWriterRAMDir.testFSDirectory fails (will be fixed)

    * TestThreadedOptimize ensureContiguousMerge fails. This one is
    a bit mysterious, perhaps the correct assertion will show where
    it's going wrong.

    I need to go through and mark the tests that can be converted to
    be NRT specific.
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 18, 2009 at 10:54 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    * TestThreadedOptimize passes, LogMergePolicy now filters the
    segmentInfos based on the dir, rather than NRTMergePolicy
    passing in only ramInfos or primaryInfos. LogMergePolicy is
    careful to select contiguous segments, by passing in a subset of
    segmentInfos, the merge policy selection broke down.

    * TestIndexWriter.testAddIndexOnDiskFull,
    testAddIndexesWithCloseNoWait fails, which I don't think
    happened before. testAddIndexOnDiskFull fails when
    autoCommit=true which I'm not sure is a valid test by the time
    this patch goes in but it probably needs to be looked into.

    The other previous notes are still valid.
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 19, 2009 at 6:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721927#action_12721927 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    Using a single segmentInfos in IW seems to be problematic as
    we'll always potentially have different dir non-contiguous
    infos. I'm seeing the error off and on in different test cases.
    I will put together a patch separating the two dir infos in IW.
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 19, 2009 at 6:58 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721933#action_12721933 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    On second thought, the previous idea would require quite a bit
    of work. Perhaps OneMerge can have the segmentInfos (ramDir or
    primaryDir) they were selected from and the ensureContiguous can
    verify that? Then we'd adjust commitMerge to remove the newly
    merged segments individually.

    I'll give this a try.
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 22, 2009 at 10:18 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    It's progressing. Randomly some tests fail such as the one noted below.

    * TestIndexWriter.testAddIndexesWithCloseNoWait fails with
    "rollback() was called or addIndexes* hit an unhandled
    exception", TestCrash.testWriterAfterCrash fails with " [junit]
    java.io.FileNotFoundException: _a.fnm [junit] at
    org.apache.lucene.store.MockRAMDirectory.openInput(MockRAMDirecto
    ry.java:252) [junit] at
    org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:67)"

    * assert in the ctor of MergeDocIDRemapper removed (not yet sure
    how to replace it)

    * OneMerge.fromInfos is added which is the set of segmentinfos
    the merge was selected from. This is for ensureContiguousMerge
    where it's failing because we have essentially two different
    sets of segmentInfos (ram and primaryDir) in the
    IW.segmentInfos. They are not related, but for convenience are
    kept together for most of IW, then are separated out in the
    merge policy. If the goal of ensureContiguousMerge is to keep
    docStoreSegments together, this will work as ramDir and
    primaryDir docStores should not need to be adjacent (I think,
    and need to verify).
    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Jun 30, 2009 at 9:29 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    Expected tests fail, the previous tests that were failing pass
    on 2 runs. I didn't make any changes though so am suspicious!

    Near Realtime Search
    --------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-dev @
categorieslucene
postedJun 1, '09 at 12:53a
activeJun 30, '09 at 9:29p
posts21
users1
websitelucene.apache.org

1 user in discussion

Jason Rutherglen (JIRA): 21 posts

People

Translate

site design / logo © 2021 Grokbase