Grokbase Groups Lucene dev April 2010
FAQ
Hi,

I've been playing around with the SpatialFilter that Chris Male and others
ares developing.
I must say it seems to work fine and I'm really eager to see the final
release.

However I think I found a small bug that occurs at scoring time in some
situations.

The SpatialFilter does its job pretty well but when

TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float
subQueryScore, float valSrcScore) {
Double docDistance = spatialFilter.getDistanceFilter().getDistance(doc);

is called I get

docDistance = null

which prevents the scoring to occur correctly.

I've been reading the code a litle to see what occurs under the hood and it
seems to me the problem is linked with the ThreadedDistanceFilter and
IndexSearcher.

The problem really occurs when IndexSearcher's public void search(Weight
weight, Filter filter, Collector collector) is called and when it has more
than 1 subReader

near line 260 of IndexSearcher

for (int i = 0; i < subReaders.length; i++) { // search each subreader
collector.setNextReader(subReaders[i], docStarts[i]);
searchWithFilter(subReaders[i], weight, filter, collector);
}



What seems to happen is that ThreadedDistanceFilter stores the correct
document id in it's inner distances Map but somehow the value that is passed
to
TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float
subQueryScore, float valSrcScore)
is not correct and probably corresponds to the bit position of the subReader
...

I hope I'm clear enough ...
:)

I have no idea on how to fix this. I hope this helps you guys and you can
find a workaround.
Let me know what I can do to fix this problem.


Thanks for helping.

Cheers,


Alex

Search Discussions

  • Uwe Schindler at Apr 25, 2010 at 8:51 am
    If you look into the source code of this CustomScore Spatial Test, there is mentioned that it is broken. The problem is that the spatial contrib. sort does not work correct with multiple segments. We are working on that, no solution here. We will have DistanceQuery in future, that does not need sorting, as it will score with the distance in adition to normal scoring.



    -----

    Uwe Schindler

    H.-H.-Meier-Allee 63, D-28213 Bremen

    http://www.thetaphi.de <http://www.thetaphi.de/>

    eMail: uwe@thetaphi.de



    From: Alex
    Sent: Sunday, April 25, 2010 4:53 AM
    To: java-dev@lucene.apache.org
    Subject: Small sorting problem in spatial contrib



    Hi,

    I've been playing around with the SpatialFilter that Chris Male and others ares developing.
    I must say it seems to work fine and I'm really eager to see the final release.

    However I think I found a small bug that occurs at scoring time in some situations.

    The SpatialFilter does its job pretty well but when

    TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float subQueryScore, float valSrcScore) {
    Double docDistance = spatialFilter.getDistanceFilter().getDistance(doc);

    is called I get

    docDistance = null

    which prevents the scoring to occur correctly.

    I've been reading the code a litle to see what occurs under the hood and it seems to me the problem is linked with the ThreadedDistanceFilter and IndexSearcher.

    The problem really occurs when IndexSearcher's public void search(Weight weight, Filter filter, Collector collector) is called and when it has more than 1 subReader

    near line 260 of IndexSearcher

    for (int i = 0; i < subReaders.length; i++) { // search each subreader
    collector.setNextReader(subReaders[i], docStarts[i]);
    searchWithFilter(subReaders[i], weight, filter, collector);
    }



    What seems to happen is that ThreadedDistanceFilter stores the correct document id in it's inner distances Map but somehow the value that is passed to
    TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float subQueryScore, float valSrcScore)
    is not correct and probably corresponds to the bit position of the subReader ...

    I hope I'm clear enough ...
    :)

    I have no idea on how to fix this. I hope this helps you guys and you can find a workaround.
    Let me know what I can do to fix this problem.


    Thanks for helping.

    Cheers,


    Alex
  • Uwe Schindler at Apr 25, 2010 at 9:23 am
    Here some links to issues, that show the problem.


    From you code it looks like you are still using 3.0.0 not 3.0.1 (as CustomScoreQuery.customScore is deprecated and fixed in 3.0.1, as this is the root of the problem): https://issues.apache.org/jira/browse/LUCENE-2190


    The new DistanceQuery is planned here:

    https://issues.apache.org/jira/browse/LUCENE-2395



    Uwe



    -----

    Uwe Schindler

    H.-H.-Meier-Allee 63, D-28213 Bremen

    <http://www.thetaphi.de/> http://www.thetaphi.de

    eMail: uwe@thetaphi.de



    From: Uwe Schindler
    Sent: Sunday, April 25, 2010 10:51 AM
    To: dev@lucene.apache.org
    Subject: RE: Small sorting problem in spatial contrib



    If you look into the source code of this CustomScore Spatial Test, there is mentioned that it is broken. The problem is that the spatial contrib. sort does not work correct with multiple segments. We are working on that, no solution here. We will have DistanceQuery in future, that does not need sorting, as it will score with the distance in adition to normal scoring.



    -----

    Uwe Schindler

    H.-H.-Meier-Allee 63, D-28213 Bremen

    http://www.thetaphi.de <http://www.thetaphi.de/>

    eMail: uwe@thetaphi.de



    From: Alex
    Sent: Sunday, April 25, 2010 4:53 AM
    To: java-dev@lucene.apache.org
    Subject: Small sorting problem in spatial contrib



    Hi,

    I've been playing around with the SpatialFilter that Chris Male and others ares developing.
    I must say it seems to work fine and I'm really eager to see the final release.

    However I think I found a small bug that occurs at scoring time in some situations.

    The SpatialFilter does its job pretty well but when

    TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float subQueryScore, float valSrcScore) {
    Double docDistance = spatialFilter.getDistanceFilter().getDistance(doc);

    is called I get

    docDistance = null

    which prevents the scoring to occur correctly.

    I've been reading the code a litle to see what occurs under the hood and it seems to me the problem is linked with the ThreadedDistanceFilter and IndexSearcher.

    The problem really occurs when IndexSearcher's public void search(Weight weight, Filter filter, Collector collector) is called and when it has more than 1 subReader

    near line 260 of IndexSearcher

    for (int i = 0; i < subReaders.length; i++) { // search each subreader
    collector.setNextReader(subReaders[i], docStarts[i]);
    searchWithFilter(subReaders[i], weight, filter, collector);
    }



    What seems to happen is that ThreadedDistanceFilter stores the correct document id in it's inner distances Map but somehow the value that is passed to
    TestSpatialFilter.DistanceCustomScoreQuery.customScore(int doc, float subQueryScore, float valSrcScore)
    is not correct and probably corresponds to the bit position of the subReader ...

    I hope I'm clear enough ...
    :)

    I have no idea on how to fix this. I hope this helps you guys and you can find a workaround.
    Let me know what I can do to fix this problem.


    Thanks for helping.

    Cheers,


    Alex
  • Alex at Apr 26, 2010 at 12:06 am
    Thank you Uwe.

    I'll check out latest version of Lucene and see if I can find a temporary
    fix with the neew CustomScoreQuery

    Cheers,

    Alex

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieslucene
postedApr 25, '10 at 2:53a
activeApr 26, '10 at 12:06a
posts4
users2
websitelucene.apache.org

2 users in discussion

Alex: 2 posts Uwe Schindler: 2 posts

People

Translate

site design / logo © 2021 Grokbase