FAQ
Hi All,

I am using Lucene 3.1 in the project.

*Background for the question:*
I am working on the application which starts with 2 threads, one performs
indexing activity and other performs searching activity (I create searcher
object from reader object). Both these threads run periodically and index or
search new documents if they are available.
My searcher searches a string in the lucene index using a BooleanQuery with
SHOULD clause.


*Problem:*
My problem is as follows:

As I see there are 2 ways to create indexReader (to create searcher)
A. indexWriter.getReader() --> to get near real time searching
or
B. Using directly the index directory path like indexReader =
IndexReader.open(NIOFSDirectory.open(new File(DIR_PATH)), true)

Now when the application starts, since there is no index created yet if I
use option "B" to create indexReader, I fail with IndexNotFoundException.
If I create IndexReader using option "A", I dont fail with exception but do
not get any result for my input which is ok since I dont have searchable
index yet.

My problem is, with option "A" of creating reader object, even after
indexing say 1000 documents and I still dont match any result. (The index
does have matching results. I can say that for sure because if I stop my
application, change the code to create reader using option B and then search
the same string on the newly created index of 1000 documents, I get lot of
matches.).

I am missing something due to which I am not getting any match for option A?
What am I doing wrong?


thanks

Saurabh

Search Discussions

  • Simon Willnauer at Jul 20, 2011 at 10:03 pm

    On Wed, Jul 20, 2011 at 11:50 PM, Saurabh Gokhale wrote:
    Hi All,

    I am using Lucene 3.1 in the project.

    *Background for the question:*
    I am working on the application which starts with 2 threads, one performs
    indexing activity and other performs searching activity (I create searcher
    object from reader object). Both these threads run periodically and index or
    search new documents if they are available.
    My searcher searches a string in the lucene index using a BooleanQuery with
    SHOULD clause.


    *Problem:*
    My problem is as follows:

    As I see there are 2 ways to create indexReader (to create searcher)
    A. indexWriter.getReader() --> to get near real time searching
    or
    B. Using directly the index directory path like indexReader =
    IndexReader.open(NIOFSDirectory.open(new File(DIR_PATH)), true)

    Now when the application starts, since there is no index created yet if I
    use option "B" to create indexReader, I fail with IndexNotFoundException.
    If I create IndexReader using option "A", I dont fail with exception but do
    not get any result for my input which is ok since I dont have searchable
    index yet.

    My problem is, with option "A" of creating reader object, even after
    indexing say 1000 documents and I still dont match any result. (The index
    does have matching results. I can say that for sure because if I stop my
    application, change the code to create reader using option B and then search
    the same string on the newly created index of 1000 documents, I get lot of
    matches.).

    I am missing something due to which I am not getting any match for option A?
    What am I doing wrong?
    hi there,

    the answer is yes, you are missing something :). When you open the
    reader from your writer via IW#getReader() you get a "current view" of
    your index (same is true when opening from directory). To refresh this
    view you need to call IndexReader#reopen() in order to seen any
    updates. For the NRT reader (created via IW#getReader()) this will be
    an uncommitted view giving you access to documetns not yet committed.
    If you are opened from a Directory you see all the already committed
    changes.

    you should use the reopen like this:
    IndexReader reader = ...
    IndexReader newReader = r.reopen();
    if (newReader != reader) {
    // reader was reopened
    reader.close();
    }
    reader = newReader;

    please be careful with reopening the reader, this operation can be
    costly if you are useing NRT readers. You should maybe do this
    periodically or if you don't have any freshness constraints simply use
    a reader opened from a directoy and reopen after commit.

    simon

    thanks

    Saurabh
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Jul 21, 2011 at 1:14 pm
    One small correction here: NRT reopen is much cheaper than IW.commit /
    IR.reopen, for getting indexed changes visible to your reader, since
    it doesn't invoke fsync and doesn't have to write deleted docs to the
    directory only to then load them again in the reader.

    But, it's still fairly costly, so you shouldn't, say, reopen your NRT
    reader for every query, if you have a high query rate; it's better to
    rate limit the reopens.

    Mike McCandless

    http://blog.mikemccandless.com

    On Wed, Jul 20, 2011 at 6:02 PM, Simon Willnauer
    wrote:
    On Wed, Jul 20, 2011 at 11:50 PM, Saurabh Gokhale
    wrote:
    Hi All,

    I am using Lucene 3.1 in the project.

    *Background for the question:*
    I am working on the application which starts with 2 threads, one performs
    indexing activity and other performs searching activity (I create searcher
    object from reader object). Both these threads run periodically and index or
    search new documents if they are available.
    My searcher searches a string in the lucene index using a BooleanQuery with
    SHOULD clause.


    *Problem:*
    My problem is as follows:

    As I see there are 2 ways to create indexReader (to create searcher)
    A. indexWriter.getReader() --> to get near real time searching
    or
    B. Using directly the index directory path like indexReader =
    IndexReader.open(NIOFSDirectory.open(new File(DIR_PATH)), true)

    Now when the application starts, since there is no index created yet if I
    use option "B" to create indexReader, I fail with IndexNotFoundException.
    If I create IndexReader using option "A", I dont fail with exception but do
    not get any result for my input which is ok since I dont have searchable
    index yet.

    My problem is, with option "A" of creating reader object, even after
    indexing say 1000 documents and I still dont match any result. (The index
    does have matching results. I can say that for sure because if I stop my
    application, change the code to create reader using option B and then search
    the same string on the newly created index of 1000 documents, I get lot of
    matches.).

    I am missing something due to which I am not getting any match for option A?
    What am I doing wrong?
    hi there,

    the answer is yes, you are missing something :). When you open the
    reader from your writer via IW#getReader() you get a "current view" of
    your index (same is true when opening from directory). To refresh this
    view you need to call IndexReader#reopen() in order to seen any
    updates. For the NRT reader (created via IW#getReader()) this will be
    an uncommitted view giving you access to documetns not yet committed.
    If you are opened from a Directory you see all the already committed
    changes.

    you should use the reopen like this:
    IndexReader reader = ...
    IndexReader newReader = r.reopen();
    if (newReader != reader) {
    // reader was reopened
    reader.close();
    }
    reader = newReader;

    please be careful with reopening the reader, this operation can be
    costly if you are useing NRT readers. You should maybe do this
    periodically or if you don't have any freshness constraints simply use
    a reader opened from a directoy and reopen after commit.

    simon

    thanks

    Saurabh
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJul 20, '11 at 9:50p
activeJul 21, '11 at 1:14p
posts3
users3
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase