searching for C++

I don't think there is a simpler way. I think you will have to modify
the tokenizer. Once you go beyond basic human-readable text, you always
end up having to do that. I have modified the JavaCC version of
StandardTokenizer for allowing symbols to pass through, but I've never
used the JFlex version - don't know anything about JFlex I'm afraid!

A good strategy might be to make a new type of lexical token called
"SYMBOL" and try to catch as many symbols as you can think of; then
maybe create new token types which are ALPHANUM types that can have
pre-fixed or post-fixed symbols.

That way, you'll be able to catch things like "c++" in a TokenFilter,
and you can choose to pass it through as a single token, or split it up
into two tokens, or whatever you want.

Hope that helps.

Regards,
JB

Alex Soto wrote:

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

This isn't ideal, but if you have a defined list of such terms, you
may find it easier to filter these terms out into a separate field
for indexing.

-h
----------------------------------------------------------------------
Hira, N.R.
Solutions Architect
Cognocys, Inc.
(773) 251-7453




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Thanks everyone. I appreciate the help.

I think I will write my own tokenizer, because I do not have a
predefined list of words with symbols.
I will modify the grammar by defining a SYMBOL token as John suggested
and redefine ALPHANUM to include it.

Regards,
Alex Soto