FAQ
[ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1313:
-------------------------------------

Attachment: LUCENE-1313.patch

Realizing the previous patches approach has grown too
complicated, this is a far simpler implementation that fulfills
the same goal, batching segments in RAM until they exceed a
given maximum size, then merging those RAM segments to a primary
directory (i.e. disk). All the while allowing all segments to be
searchable with a minimum latency turnaround.

* Segment names are generated for the ram writer from the
primary writer, this insures name continuity. Actually I'm not
sure this is necessary anymore.

* The problem is when the ram segments are merged into the
primary writer, they appear to be non-contiguous. Some of the
contiguous segment checking has been relaxed for this case, and
needs to be conditional on the segment merging being from the
ram dir. Perhaps we can have our cake and eat it too here by
keeping the contiguous check around for all cases?

* When the ram writer's usage exceeds a specified size, the ram
buffer is flushed, and the ram segments are synchronously merged
to the primary writer using a mechanism similar to
addIndexesNoOptimize.
Near Realtime Search (using a built in RAMDirectory)
----------------------------------------------------

Key: LUCENE-1313
URL: https://issues.apache.org/jira/browse/LUCENE-1313
Project: Lucene - Java
Issue Type: New Feature
Components: Index
Affects Versions: 2.4.1
Reporter: Jason Rutherglen
Priority: Minor
Fix For: 3.1

Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


Enable near realtime search in Lucene without external
dependencies. When RAM NRT is enabled, the implementation adds a
RAMDirectory to IndexWriter. Flushes go to the ramdir unless
there is no available space. Merges are completed in the ram
dir until there is no more available ram.
IW.optimize and IW.commit flush the ramdir to the primary
directory, all other operations try to keep segments in ram
until there is no more space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Search Discussions

  • Jason Rutherglen (JIRA) at Nov 3, 2009 at 12:39 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    * Ensure contiguous is mostly back

    * Cleaned up the code and made the flush method non-synchronized

    * There's a subtle synchronization bug causing files to not be found in the testRandomThreads method

    * There's excessive merge logging to debug the sync issue
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 4, 2009 at 9:41 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    I wanted to simplify a little more to more easily understand the
    edge cases that fail. In the multithreaded test, files were
    sometimes left open which is hard for me to debug.

    The TestNRT.testSimple method passes however, IndexFileDeleter
    is complaining about not being able to delete when expected
    which is shown in the IW.infoStream.

    The NRT.flush method creates a merge of all the ram segments,
    then calls IW.mergeIn to manually merge the ram segments into
    the writer. OneMerge contains the writer where the segment
    readers should be obtained from. In this case, the primary
    writer obtains the readers from the ram writer's readerpool.
    This is important because deletes may be coming in as we're
    merging. However I'm not sure this will work without a shared
    lock between the writers for commitMergedDeletes which requires
    syncing.

    Mike, can you take a look to see if this path will work?
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 2:05 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    The tests pass, and the previous kinks seem to be worked out.
    Actually there is still one issue, where in waitForMerges, the
    assert mergingSegments size equals zero occasionally fails. I
    think this is a small sync problem because of the manual merge
    between the two writers.

    I'll run the multi threaded test at a longer interval to see
    what other errors may crop up. Once it runs successfully for
    lets say 30 minutes, we can beef up the stress testing of this
    patch by doing concurrent updates, deletes, etc.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 2:15 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    Alright, the issue was simple, OneMerge.registerDone was being set to false by the primary writer, so the ram writer wasn't removing the infos from mergingSegments in mergeFinish.


    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 6:34 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    TestNRTReaderWithThreads2 fails periodically, it's just another
    synchronization issue. I added syncing on the merge writer in
    methods like commitMergedDeletes and commitMerge. Perhaps more
    of that type of syncing needs to be added. It can take time for
    these issues to be figured out.

    There's also remnants of a first attempt at transparently
    utilizing the NRT class within IW.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 7:34 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    New assertions in NRT.flush are catching the issue that's occurring.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Nov 5, 2009 at 7:45 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774049#action_12774049 ]

    Michael McCandless commented on LUCENE-1313:
    --------------------------------------------

    I'll try to have a look at this soon Jason! Sounds like good progress...
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 9:45 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    OK, all the tests pass consistently now.

    I guess the next feature is to have NRT.flush execute in a single background thread rather than block update doc calls.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 5, 2009 at 9:57 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774098#action_12774098 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    I turned on assert !sr.hasChanges in readerPool.release, and it fails sometimes. I'm not quite sure why yet.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 6, 2009 at 1:24 am
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774171#action_12774171 ]

    Jason Rutherglen commented on LUCENE-1313:
    ------------------------------------------

    Well, I added the background thread for NRT.flush, however, I've also been debugging this assert !sr.hasChanges issue, which out of 7000 iterations, occurs once, and is fairly minor. Hmm... Apply deletes shouldn't really conflict so I'm hoping this isn't an original bug unrelated to LUCENE-1313.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 9, 2009 at 8:48 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    This patch includes flushing in a background thread. Some formatting has been cleaned up, javadocs added.

    I ran TestNRTReaderWithThreads2 a couple times for kicks and didn't see the assert sr.hasChanges error. I'll probably focus on adding more stress testing.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org
  • Jason Rutherglen (JIRA) at Nov 24, 2009 at 11:56 pm
    [ https://issues.apache.org/jira/browse/LUCENE-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Jason Rutherglen updated LUCENE-1313:
    -------------------------------------

    Attachment: LUCENE-1313.patch

    I went back to trying to utilize a RAM dir inside of IW. This
    actually works well now, and the code is less intrusive than the
    previous patches. The incoming directory is placed in a
    PrefixSwitchDirectory which accepts indicating whether a segment
    is destined for the ram directory or the primary directory.

    A single internal segment infos collection is used, the first
    half are primary dir segments, the second, ram dir segments.

    The above mentioned changes of course break many unit tests. I'm
    going through and evaluating what do on a case by case basis,
    and am open to suggestions.
    Near Realtime Search (using a built in RAMDirectory)
    ----------------------------------------------------

    Key: LUCENE-1313
    URL: https://issues.apache.org/jira/browse/LUCENE-1313
    Project: Lucene - Java
    Issue Type: New Feature
    Components: Index
    Affects Versions: 2.4.1
    Reporter: Jason Rutherglen
    Priority: Minor
    Fix For: 3.1

    Attachments: LUCENE-1313.jar, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, LUCENE-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch, lucene-1313.patch


    Enable near realtime search in Lucene without external
    dependencies. When RAM NRT is enabled, the implementation adds a
    RAMDirectory to IndexWriter. Flushes go to the ramdir unless
    there is no available space. Merges are completed in the ram
    dir until there is no more available ram.
    IW.optimize and IW.commit flush the ramdir to the primary
    directory, all other operations try to keep segments in ram
    until there is no more space.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-dev-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-dev @
categorieslucene
postedNov 2, '09 at 7:05p
activeNov 24, '09 at 11:56p
posts13
users1
websitelucene.apache.org

1 user in discussion

Jason Rutherglen (JIRA): 13 posts

People

Translate

site design / logo © 2021 Grokbase