StandardAnalyzer and thread safety

Hi,
I've been reading my new project bible 'Lucene in Action' about Analysis in
Chapter 4 and wondered what others are doing for indexing XML(if anyone else
is, that is!).
Are you folks just writing your own or utilising the current Lucene analysis
libraries?
thanks,
Malcolm Clark


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Analyzers are at a per-field granularity, and more than likely your
XML data contains what you would want treated as multiple fields. So
while an analyzer _could_ directly deal with XML, it really is
unlikely to be the appropriate layer to do so. The majority of
scenarios would have XML parsed separately and then the individual
separated text fed to Lucene fields for analysis.

Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Not exactly sure what you're asking with regards to Analyzers and
parsing XML...

But for parsing and indexing XML documents with Lucene, you can find a
lot of material out there by searching the list archives and using
google. However, the document I found most helpful was this piece
written by Otis:

http://www-128.ibm.com/developerworks/java/library/j-lucene/

I primarily used that to get me started with using Digester and Lucene
together to index XML documents. You can download Apache Commons
Digester from here:

http://jakarta.apache.org/commons/digester/

Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Hi,
I'm just asking for opinions on Analyzer's for the indexing. For example
Otis in his article uses the WhitespaceAnalyzer and the Sandbox program uses
the StandardAnalyzer.I am just gauging opinions on the subject with regard
to XML.
I'm using a mix of the Sandbox XMLDocumentHandlerSAX and a bit extra. I
originally started using Digester but found that I preferred the Sandbox
implementation.
Thanks,
Malcolm Clark


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Not sure I am understanding your question correctly, but I think you
want to pick your Analyzer based on what is in your content (i.e.
language, usage of special symbols, etc.), not based on what the format
of your content is (i.e. XML).

Malcolm wrote:

Ok... just got confused because you mentioned XML. Unless you're
actually indexing the raw XML in some of your fields, the fact that
you're indexing XML documents as your source content is irrelevant to
your choice of Analyzer.

Choice of indexer really depends on your specific project requirements
and what level of querying functionality your client needs. For example,
I started off using the StandardAnalyzer because it incorporates some
very useful and sophisticated functionality. But I found that it was
removing many stop words that my client requested the ability to query
with, so I will end up using my own custom analyzer class primarily
based on the StandardAnalyzer but modifying the stop word list.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

I'm currently indexing the INEX collection and then performing queries on
the Format features within the text. I am using a wide range of the XML
features. The reason I asked about the XML Analysis is I am interesting in
opinions and reasons for adding a wide range of discussion to my
dissertation regarding Lucene.
Thanks,
Malcolm


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]