FAQ
In the current version of lucene, 3.0 the following methods are no longer
available.
- TokenStream.next()
- TokenStream.next(Token).
- Token.setTermText()
- Token.termText().

The newer versions says to use, incrementToken() and AttributeSource APIs.
But I cannot find much help using these in this way. Any help again is
appreciated. If anyone has a basic example, or can point me to something
useful that would be awesome. Thanx.

Merry Christmas too.

public class CustomAnalyzer extends Analyzer
{
@Override
public TokenStream tokenStream(final String fieldName, final Reader
reader)
{
TokenStream ts = new WhitespaceTokenizer(reader);
ts = new StopFilter(false, ts, StopAnalyzer.ENGLISH_STOP_WORDS_SET);
ts = new LowerCaseFilter(ts);
ts = new CustomFilter(ts);

return ts;
}

}

public class CustomFilter extends TokenFilter
{
protected CustomFilter(TokenStream tokenStream)
{
super(tokenStream);
}
@Override
public Token next(final Token reusableToken) throws IOException
{
Token nextToken = input.next(reusableToken);

if(nextToken != null)
{

nextToken.setTermText(nextToken.termText().replaceAll(":|,|\\(|\\)|“|~|;|&|\\.",""));
}
return nextToken;
}
}
--
View this message in context: http://old.nabble.com/help-customfilter-with-incrementToken%28%29-and-AttributeSource-APIs-tp26915600p26915600.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Digy at Dec 24, 2009 at 6:16 pm
    The source code for LowerCaseFilter or StopFilter can be a good starting point.
    DIGY

    -----Original Message-----
    From: maxSchlein
    Sent: Thursday, December 24, 2009 7:10 PM
    To: java-user@lucene.apache.org
    Subject: help customfilter with incrementToken() and AttributeSource APIs


    In the current version of lucene, 3.0 the following methods are no longer
    available.
    - TokenStream.next()
    - TokenStream.next(Token).
    - Token.setTermText()
    - Token.termText().

    The newer versions says to use, incrementToken() and AttributeSource APIs.
    But I cannot find much help using these in this way. Any help again is
    appreciated. If anyone has a basic example, or can point me to something
    useful that would be awesome. Thanx.

    Merry Christmas too.

    public class CustomAnalyzer extends Analyzer
    {
    @Override
    public TokenStream tokenStream(final String fieldName, final Reader
    reader)
    {
    TokenStream ts = new WhitespaceTokenizer(reader);
    ts = new StopFilter(false, ts, StopAnalyzer.ENGLISH_STOP_WORDS_SET);
    ts = new LowerCaseFilter(ts);
    ts = new CustomFilter(ts);

    return ts;
    }

    }

    public class CustomFilter extends TokenFilter
    {
    protected CustomFilter(TokenStream tokenStream)
    {
    super(tokenStream);
    }
    @Override
    public Token next(final Token reusableToken) throws IOException
    {
    Token nextToken = input.next(reusableToken);

    if(nextToken != null)
    {

    nextToken.setTermText(nextToken.termText().replaceAll(":|,|\\(|\\)|“|~|;|&|\\.",""));
    }
    return nextToken;
    }
    }
    --
    View this message in context: http://old.nabble.com/help-customfilter-with-incrementToken%28%29-and-AttributeSource-APIs-tp26915600p26915600.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedDec 24, '09 at 5:10p
activeDec 24, '09 at 6:16p
posts2
users2
websitelucene.apache.org

2 users in discussion

Digy: 1 post maxSchlein: 1 post

People

Translate

site design / logo © 2022 Grokbase