Basic question about indexing certain words

It depends completely on what analyzer you use. Conceptually, an Analyzer
is composed of a Tokenizer followed by any number of Filters. So the
input stream is broken up by the Tokenizer, then each token has one or
more Filters applied (e.g. LowerCaseFilter, StopWordFilter)..

The reason I'm not answering your question directly is that I can't. If you
choose, say, a WhitespaceAnalyzer, which is built from a
WhitespaceTokenizer,
then your hyphens and apostrophes will pass through as-is, and your tokens
(the minimal searchable unit) will be "Jane" "Doe-Smith" and "Sa'eed",
capitals
and all.

If you choose StandardAnalyzer, built on StandardTokenizer and several
filters
your tokens would be "jane" "doe" "smith" "sa" "eed" (note lower-casing as
well).

You can build your own Analyzers to process text however you please. Lucene
In Action has quite a thorough explanation of this process, you'll save
yourself
a bunch of time by reading those sections. You can get the second edition
of that book in electronic form from Manning through their early access
program.

Until you understand this process well, I'd recommend that you be very, very
sure that you use the *same* analyzer for both indexing and searching or
your
results will be...surprising.

Think about getting a copy of Luke to examine your indexes, that tool makes
it
easy to see the effects of various Analyzers. Google Lucene Luke....

Finally, you can easily use *different* analyzers for different fields
within a
document, see PerFieldAnalyzerWrapper.

HTH
Erick

Thanks very much for such a detailed reply, I didn't realize that there was
so much to this subject. I understand the issue a bit better now!

Take care.


Erick Erickson wrote: