stop words and index size

: The corpus is the English Wikipedia, and I indexed the title and body of
: the articles. I used a list of 525 stop words.
:
: With stopwords removed the index is 227MB.
: With stopwords kept the index is 331MB.

That doesn't seem horribly surprising.

consider that for every Term in the index, lucene is keeping track of the
list of <docId, freq> pairs for every document that contains that term.

Assume that something has to be in at least 25% of the docs before you
decide it's worth making it a stop word. your URL indicates you are
dealing with 400k docs, which means that for each stop word, the space
need to store the int pairs for <docId, freq> is...

(4B + 4B) * 100,000 =~ 780KB (per stop word Term, minimum)

...not counting any indexing structures that may be used internally to
improve the lookup of a Term. assuming some of those words are in more or
less then 25% of your documents, that could easily account for a
differents of 100MB.

I suspect that an interesting excersize would be to use some of the code
I've seen tossed arround on this list that lets you iterate over all Terms
and find the most common once to help you determine your stopword list
progromaticly. Then remove/reindex any documents that have each word as
you add it to your stoplist (one word at a time) and watch your index
shrink.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

The unstopped version is indeed bigger and slower to build, but it's
only slower to search when folks search on stop words. One approach to
minimizing stopwords in searches (used by, e.g. Nutch & Google) is to
index all stop words but remove them from queries unless they're (a) in
a phrase or (b) explicitly required with a "+". (It might be nice if
Lucene included a query parser that had this feature.)

Nutch also optimizes phrase searches involving a few very common stop
words (e.g., "the", "a", "to") by indexing these as bigrams and
converting phrases involving them to bigram phrases. So, if someone
searches for "to be or not to be" then this turns into a search for
"to-be be or not-to to-be" which is considerably faster since it
involves rarer terms. But the more words you bigram the bigger the
index gets and the slower updates get, so you probably can't afford to
do this for your full stop list. (It might be nice if Lucene included
support for this technique too!)

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org