I upgraded to 1.0.4 and am experiencing some strange problems with the
parsing of a number value range.

The first line is the query string being passed to the query parser, and
the second line is resulting query description.

test 20071113153701..20071114153701
Xapian::Query((Ztest:(pos=1) FILTER VALUE_RANGE 0 ?J?w)@ ?J?G?@))

I am using the Search::Xapian perl module to do the searching, and omega
scriptindex tool to build the index.

Any ideas?

- Jim

Search Discussions

  • Olly Betts at Nov 15, 2007 at 10:23 am

    On Wed, Nov 14, 2007 at 03:51:52PM -0500, Jim Spath wrote:
    The first line is the query string being passed to the query parser, and
    the second line is resulting query description.

    test 20071113153701..20071114153701
    Xapian::Query((Ztest:(pos=1) FILTER VALUE_RANGE 0 ?J?w)@ ?J?G?@))
    NumberValueRangeProcessor expects the value to be encoded with
    Xapian::sortable_serialise(). Otherwise the string compare which
    is used means that 123 < 20 < 3!

    In 1.0.0, NumberValueRangeProcessor had a broken design where it just
    used a string compare like this, but we did quickly realise and warn
    people not to use it until it was fixed.
    I am using the Search::Xapian perl module to do the searching, and omega
    scriptindex tool to build the index.
    I don't think scriptindex supports encoding with sortable_serialise()
    currently. But if your timestamps are always fixed length strings then
    you could just use StringValueRangeProcessor.

    Cheers,
    Olly
  • Jim Spath at Nov 15, 2007 at 2:20 pm

    Olly Betts wrote:
    On Wed, Nov 14, 2007 at 03:51:52PM -0500, Jim Spath wrote:
    The first line is the query string being passed to the query parser, and
    the second line is resulting query description.

    test 20071113153701..20071114153701
    Xapian::Query((Ztest:(pos=1) FILTER VALUE_RANGE 0 ?J?w)@ ?J?G?@))
    NumberValueRangeProcessor expects the value to be encoded with
    Xapian::sortable_serialise(). Otherwise the string compare which
    is used means that 123 < 20 < 3!

    In 1.0.0, NumberValueRangeProcessor had a broken design where it just
    used a string compare like this, but we did quickly realise and warn
    people not to use it until it was fixed.
    I am using the Search::Xapian perl module to do the searching, and omega
    scriptindex tool to build the index.
    I don't think scriptindex supports encoding with sortable_serialise()
    currently. But if your timestamps are always fixed length strings then
    you could just use StringValueRangeProcessor.
    StringValueRangeProcessor works for me!

    Thanks again Olly.
  • Adam Sjøgren at Dec 20, 2007 at 2:56 pm

    On Thu, 15 Nov 2007 10:23:22 +0000, Olly wrote:
    On Wed, Nov 14, 2007 at 03:51:52PM -0500, Jim Spath wrote:

    test 20071113153701..20071114153701
    Xapian::Query((Ztest:(pos=1) FILTER VALUE_RANGE 0 ?J?w)@ ?J?G?@))
    NumberValueRangeProcessor expects the value to be encoded with
    Xapian::sortable_serialise(). Otherwise the string compare which
    is used means that 123 < 20 < 3!
    Is Xapian::sortable_serialise() wrapped in Search::Xapian?

    It didn't seem so when I tried it just now (1.0.4.0-2 from Debian
    unstable) - which means that NumberValueRangeProcessor is not that
    usable if you index from Perl, right?
    I am using the Search::Xapian perl module to do the searching, and omega
    scriptindex tool to build the index.
    I don't think scriptindex supports encoding with sortable_serialise()
    currently. But if your timestamps are always fixed length strings then
    you could just use StringValueRangeProcessor.
    I was trying to use StringValueRangeProcessor, but the constructor does
    not take a prefix (as opposed to NumberVRP), so I can only have one
    "rangeable" value, which is too little in my case.

    Should I try figuring out how to wrap sortable_serialise, or should I
    try to provide a patch for StringVRP that adds the str and prefix
    arguments, similar to NumberVRP?


    Best regards,

    Adam

    --
    "Your father traded the tools for M&Ms again" Adam Sj?gren
    asjo@koldfront.dk
  • Olly Betts at Dec 20, 2007 at 4:36 pm

    On Thu, Dec 20, 2007 at 03:56:43PM +0100, Adam Sj?gren wrote:
    On Thu, 15 Nov 2007 10:23:22 +0000, Olly wrote:

    NumberValueRangeProcessor expects the value to be encoded with
    Xapian::sortable_serialise(). Otherwise the string compare which
    is used means that 123 < 20 < 3!
    Is Xapian::sortable_serialise() wrapped in Search::Xapian?
    Not currently I'm afraid.
    It didn't seem so when I tried it just now (1.0.4.0-2 from Debian
    unstable)
    Odd, Debian only seems to have -1 currently:

    http://packages.qa.debian.org/libs/libsearch-xapian-perl.html

    Perhaps you're using a privately updated package? But these issues
    affect vanilla 1.0.4.0 anyway.
    - which means that NumberValueRangeProcessor is not that
    usable if you index from Perl, right? Indeed.
    I was trying to use StringValueRangeProcessor, but the constructor does
    not take a prefix (as opposed to NumberVRP), so I can only have one
    "rangeable" value, which is too little in my case.
    StringVRP really ought to support a prefix/suffix like NumberVRP does.
    There's already a wishlist bug for adding this feature to DateVRP, so
    I've updated that to cover StringVRP too:

    http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id"0
    Should I try figuring out how to wrap sortable_serialise, or should I
    try to provide a patch for StringVRP that adds the str and prefix
    arguments, similar to NumberVRP?
    Patches for either or both would certainly be welcome. I'm currently
    busy trying to get a 1.0.5 release together.

    Cheers,
    Olly
  • Adam Sjøgren at Dec 20, 2007 at 5:33 pm

    On Thu, 20 Dec 2007 16:36:02 +0000, Olly wrote:
    On Thu, Dec 20, 2007 at 03:56:43PM +0100, Adam Sj?gren wrote:

    Is Xapian::sortable_serialise() wrapped in Search::Xapian?
    Not currently I'm afraid.
    Ok - I couldn't quite figure out where it would go, by my patented
    monkey-see - monkey-copy method.
    It didn't seem so when I tried it just now (1.0.4.0-2 from Debian
    unstable)
    Odd, Debian only seems to have -1 currently:
    Ah - I accidentally bumped the packaging number to -2 when I built a
    local backport to Ubuntu gutsy; I should have said 1.0.4.0-1, sorry.
    Perhaps you're using a privately updated package? But these issues
    affect vanilla 1.0.4.0 anyway.
    Yes - the only thing I updated was the debian/changelog (so we can
    easily distinguish locally built packages).
    StringVRP really ought to support a prefix/suffix like NumberVRP does.
    There's already a wishlist bug for adding this feature to DateVRP, so
    I've updated that to cover StringVRP too:
    http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id"0
    Cool.
    Should I try figuring out how to wrap sortable_serialise, or should I
    try to provide a patch for StringVRP that adds the str and prefix
    arguments, similar to NumberVRP?
    Patches for either or both would certainly be welcome. I'm currently
    busy trying to get a 1.0.5 release together.
    My C is rusty and my C++ has never really been shining, but I will see
    if I can find time to have a go at the latter tomorrow; I think we need
    it for our app, so I am motivated :-)


    Best regards,

    Adam

    --
    "But they are all following you! Adam Sj?gren
    No, they ain't, I'm just in front" asjo@koldfront.dk
  • Ron Kass at Dec 20, 2007 at 10:09 pm
    We wrapped it for our own needs for Perl usage.

    Do the following:

    in the XS directory, there is a file called Database.xs, towards the end
    of it add the following :

    string
    Database::sortable_serialise(double value)
    CODE:
    try {
    RETVAL = sortable_serialise(value);
    }
    catch (const Error &error) {
    croak( "Exception: %s", error.get_msg().c_str() );
    }
    OUTPUT:
    RETVAL

    double
    Database::sortable_unserialise(string value)
    CODE:
    try {
    RETVAL = sortable_unserialise(value);
    }
    catch (const Error &error) {
    croak( "Exception: %s", error.get_msg().c_str() );
    }
    OUTPUT:
    RETVAL

    Note: Place this code just BEFORE

    void
    Database::DESTROY()


    Then recompile the Perl-XS (follow the readme file)..

    You will now be able to call sortable_serialise (and
    sortable_unserialise) from perl, via any database object.


    For example: (just a general example)

    my $XAP_DB Search::Xapian::WritableDatabase->new("/test_database/",DB_CREATE_OR_OPEN);


    my $TimeStamp = 1035122221;

    my $TimeStamp_Serialized = $XAP_DB->sortable_serialise($TimeStamp);


    This will return the funky string thingy you saw in the query, so you
    can now store those as values and use as intended..

    In this example we use an actual WritableDatabase, but it can be any
    database.

    Note that we did this per our need, and it may very well be not the
    perfect/ideal wrapping, but it works perfectly for us. I think the best
    place to wrap it is part of the actual top Xapian object, but we didn't
    succeed with it right away and it was easier (and enough) for us to do
    on the Database object, so thats what we kept for now...


    Hope it helps. Good luck..


    Ron


    Adam Sj?gren wrote:
    On Thu, 20 Dec 2007 16:36:02 +0000, Olly wrote:

    On Thu, Dec 20, 2007 at 03:56:43PM +0100, Adam Sj?gren wrote:
    Is Xapian::sortable_serialise() wrapped in Search::Xapian?
    Not currently I'm afraid.
    Ok - I couldn't quite figure out where it would go, by my patented
    monkey-see - monkey-copy method.

    It didn't seem so when I tried it just now (1.0.4.0-2 from Debian
    unstable)
    Odd, Debian only seems to have -1 currently:
    Ah - I accidentally bumped the packaging number to -2 when I built a
    local backport to Ubuntu gutsy; I should have said 1.0.4.0-1, sorry.

    Perhaps you're using a privately updated package? But these issues
    affect vanilla 1.0.4.0 anyway.
    Yes - the only thing I updated was the debian/changelog (so we can
    easily distinguish locally built packages).

    StringVRP really ought to support a prefix/suffix like NumberVRP does.
    There's already a wishlist bug for adding this feature to DateVRP, so
    I've updated that to cover StringVRP too:
    http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id"0
    Cool.

    Should I try figuring out how to wrap sortable_serialise, or should I
    try to provide a patch for StringVRP that adds the str and prefix
    arguments, similar to NumberVRP?
    Patches for either or both would certainly be welcome. I'm currently
    busy trying to get a 1.0.5 release together.
    My C is rusty and my C++ has never really been shining, but I will see
    if I can find time to have a go at the latter tomorrow; I think we need
    it for our app, so I am motivated :-)


    Best regards,

    Adam
  • Adam Sjøgren at Dec 21, 2007 at 12:01 pm

    On Fri, 21 Dec 2007 00:09:06 +0200, Ron wrote:

    We wrapped it for our own needs for Perl usage.
    Do the following:
    [... nice description elided ...]

    Cool - thanks! How about contributing a patch, so it can be integrated
    into Xapian proper?


    Best regards,

    Adam

    --
    "Computer science is no more about computers than Adam Sj?gren
    astronomy is about telescopes." asjo@koldfront.dk

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupxapian-discuss @
categoriesxapian
postedNov 14, '07 at 8:51p
activeDec 21, '07 at 12:01p
posts8
users4
websitexapian.org
irc#xapian

People

Translate

site design / logo © 2021 Grokbase