Grokbase › Groups › Lucene › java-user › May 2009
FAQ
Hello fellow Lucene developers!

I have a bit of a question - and I can't find the answer in my lucene
book....

Im trying to create a query that will query 2 fields using different
analyzers and combine the scores together to give me my "hits". The idea is
that for the one dataset I want a pure text match only, and for the other I
want to use the stemming concept by using a custom made analyzer.

Is there a way to do this?

--This is what I am thinking (conceptually)------
MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
StandardAnalyzer(), boosts);
MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
PositionalPorterStopAnalyzer(), boosts);

my MultiFieldQueryParser = mfqp1 + mfqp2
----------------

The issue that I see is that if I just use one analyzer, I lose out. I know
I need to query multiple fields, in multiple ways, but I just dont know how
to make this work....

Any ideas?
--
View this message in context: http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23338538.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Erick Erickson at May 1, 2009 at 9:51 pm ⇧
    This looks like a job for PerFieldAnalyzerWrapper, no
    MultiFieldQueryparser required........

    Best
    Erick
    On Fri, May 1, 2009 at 3:33 PM, theDude_2 wrote:


    Hello fellow Lucene developers!

    I have a bit of a question - and I can't find the answer in my lucene
    book....

    Im trying to create a query that will query 2 fields using different
    analyzers and combine the scores together to give me my "hits". The idea
    is
    that for the one dataset I want a pure text match only, and for the other I
    want to use the stemming concept by using a custom made analyzer.

    Is there a way to do this?

    --This is what I am thinking (conceptually)------
    MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
    StandardAnalyzer(), boosts);
    MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
    PositionalPorterStopAnalyzer(), boosts);

    my MultiFieldQueryParser = mfqp1 + mfqp2
    ----------------

    The issue that I see is that if I just use one analyzer, I lose out. I
    know
    I need to query multiple fields, in multiple ways, but I just dont know how
    to make this work....

    Any ideas?
    --
    View this message in context:
    http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23338538.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Uwe Schindler at May 1, 2009 at 9:57 pm ⇧
    Hi Erick,

    this is not what he wants to do, I think.

    He wants both things at once: One query string that searches both fields
    with different analyzers. For searching more than one field at once, he can
    use MultiFieldQueryParser, but not if the analyzer for both fields is
    different (because the query string itself is also analyzed, but the two
    different analyzers will produce different tokens which cannot be directly
    searched in both fields). So the query string should produce two different
    anaylzed tokens and the search for each token group should be done in
    parallel.

    -----
    Uwe Schindler
    H.-H.-Meier-Allee 63, D-28213 Bremen
    http://www.thetaphi.de
    eMail: uwe@thetaphi.de
    -----Original Message-----
    From: Erick Erickson
    Sent: Friday, May 01, 2009 11:42 PM
    To: java-user@lucene.apache.org
    Subject: Re: MultiFieldQueryParser - using a different analyzer per
    field...

    This looks like a job for PerFieldAnalyzerWrapper, no
    MultiFieldQueryparser required........

    Best
    Erick
    On Fri, May 1, 2009 at 3:33 PM, theDude_2 wrote:


    Hello fellow Lucene developers!

    I have a bit of a question - and I can't find the answer in my lucene
    book....

    Im trying to create a query that will query 2 fields using different
    analyzers and combine the scores together to give me my "hits". The idea
    is
    that for the one dataset I want a pure text match only, and for the other I
    want to use the stemming concept by using a custom made analyzer.

    Is there a way to do this?

    --This is what I am thinking (conceptually)------
    MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
    StandardAnalyzer(), boosts);
    MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
    PositionalPorterStopAnalyzer(), boosts);

    my MultiFieldQueryParser = mfqp1 + mfqp2
    ----------------

    The issue that I see is that if I just use one analyzer, I lose out. I
    know
    I need to query multiple fields, in multiple ways, but I just dont know how
    to make this work....

    Any ideas?
    --
    View this message in context:
    http://www.nabble.com/MultiFieldQueryParser---using-a-different-
    analyzer-per-field...-tp23338538p23338538.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Erick Erickson at May 2, 2009 at 12:25 am ⇧
    Hmmmm... If that's the case, would you suggest copying the
    relevant fields, using the different analyzers at index time? Then perhaps
    combining the two Queries returned from the MFQP into two clauses
    in a BooleanQuery? Or is there a more elegant way?

    Because you're right of course, trying to use two different
    Analyzers on the same field(s) would produce...er..."interesting" results.

    Best
    Erick
    On Fri, May 1, 2009 at 5:57 PM, Uwe Schindler wrote:

    Hi Erick,

    this is not what he wants to do, I think.

    He wants both things at once: One query string that searches both fields
    with different analyzers. For searching more than one field at once, he can
    use MultiFieldQueryParser, but not if the analyzer for both fields is
    different (because the query string itself is also analyzed, but the two
    different analyzers will produce different tokens which cannot be directly
    searched in both fields). So the query string should produce two different
    anaylzed tokens and the search for each token group should be done in
    parallel.

    -----
    Uwe Schindler
    H.-H.-Meier-Allee 63, D-28213 Bremen
    http://www.thetaphi.de
    eMail: uwe@thetaphi.de
    -----Original Message-----
    From: Erick Erickson
    Sent: Friday, May 01, 2009 11:42 PM
    To: java-user@lucene.apache.org
    Subject: Re: MultiFieldQueryParser - using a different analyzer per
    field...

    This looks like a job for PerFieldAnalyzerWrapper, no
    MultiFieldQueryparser required........

    Best
    Erick
    On Fri, May 1, 2009 at 3:33 PM, theDude_2 wrote:


    Hello fellow Lucene developers!

    I have a bit of a question - and I can't find the answer in my lucene
    book....

    Im trying to create a query that will query 2 fields using different
    analyzers and combine the scores together to give me my "hits". The idea
    is
    that for the one dataset I want a pure text match only, and for the other I
    want to use the stemming concept by using a custom made analyzer.

    Is there a way to do this?

    --This is what I am thinking (conceptually)------
    MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
    StandardAnalyzer(), boosts);
    MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
    PositionalPorterStopAnalyzer(), boosts);

    my MultiFieldQueryParser = mfqp1 + mfqp2
    ----------------

    The issue that I see is that if I just use one analyzer, I lose out. I
    know
    I need to query multiple fields, in multiple ways, but I just dont know how
    to make this work....

    Any ideas?
    --
    View this message in context:
    http://www.nabble.com/MultiFieldQueryParser---using-a-different-
    analyzer-per-field...-tp23338538p23338538.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • theDude_2 at May 4, 2009 at 1:24 pm ⇧
    Hey guys: original poster here, and I found a solution! I created a wrapper
    that could accept multiple analyzers and then combined them into a search:
    here is the code.

    ------wrapper class---------

    public class PositionalPorterStopAnalyzer extends Analyzer {
    private Set stopWords;
    public PositionalPorterStopAnalyzer() {
    this(StopAnalyzer.ENGLISH_STOP_WORDS);
    }
    public PositionalPorterStopAnalyzer(String[] stopList) {
    stopWords = StopFilter.makeStopSet(stopList);
    }
    public TokenStream tokenStream(String fieldName, Reader reader) {
    StopFilter stopFilter = new StopFilter(new LowerCaseTokenizer(reader),
    stopWords);
    stopFilter.setEnablePositionIncrements(true);
    return new PorterStemFilter(stopFilter);
    }
    }

    ------creating the search---------

    PerFieldAnalyzerWrapper pfawBoth = new PerFieldAnalyzerWrapper(
    new StandardAnalyzer());
    pfawBoth.addAnalyzer("Field_1", new StandardAnalyzer());
    pfawBoth.addAnalyzer("Field_2", new PositionalPorterStopAnalyzer());

    mfqp = new MultiFieldQueryParser(
    fields,
    pfawBoth,
    boosts);

    -----------

    thanks for all the help


    ----------------

    theDude_2 wrote:
    Hello fellow Lucene developers!

    I have a bit of a question - and I can't find the answer in my lucene
    book....

    Im trying to create a query that will query 2 fields using different
    analyzers and combine the scores together to give me my "hits". The idea
    is that for the one dataset I want a pure text match only, and for the
    other I want to use the stemming concept by using a custom made analyzer.

    Is there a way to do this?

    --This is what I am thinking (conceptually)------
    MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
    StandardAnalyzer(), boosts);
    MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
    PositionalPorterStopAnalyzer(), boosts);

    my MultiFieldQueryParser = mfqp1 + mfqp2
    ----------------

    The issue that I see is that if I just use one analyzer, I lose out. I
    know I need to query multiple fields, in multiple ways, but I just dont
    know how to make this work....

    Any ideas?
    --
    View this message in context: http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23367209.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedMay 1, '09 at 7:34p
activeMay 4, '09 at 1:24p
posts5
users3
websitelucene.apache.org

3 users in discussion

theDude_2: 2 posts Erick Erickson: 2 posts Uwe Schindler: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase