Sorry, the image wasn't sent:

http://wwwhome.cs.utwente.nl/~trieschn/bm25.PNG-----Original Message-----

From: Trieschnigg, R.B. (Dolf)

Sent: vrijdag 17 februari 2006 10:54

To:

[email protected]Subject: RE: BM25 Similarity implementation

I would like to implement the Okapi BM25 weighting

function using my

own Similarity implementation. Unfortunately BM25 requires the

document length in the score calculation, which is not

provided by

the Scorer.

How do you want to measure document length? If the number of tokens

is an acceptable measure, then the norm contains

sqrt(numTokens) by default. You can modify your

Similarity.lengthNorm() implementation to not perform the sqrt, or

square the norm.

I assume the number of tokens will be a good estimate.

I've included an image with the algorithm (my ASCII art isn't

that good).

Legend of the figure:

- k1, k3 and b are constants

- tf is the within document term frequency

- df is the document frequency

- N is the collection size

- r is the number of relevant documents containing a

particular term (without relevance information assumed to be 0)

- R is the number of items known to be relevant to a specific

topic (without relevance information assumed to be 0)

As far is I understand Lucene multiplies the squared weight

with the result of Similarity.lengthNorm(), but BM25 requires

the document length for the calculation of the document term

weighting (as far as I know it's not possible to extract the

influence of the normalization as a constant multiplier).

Am I missing something here?

Dolf

---------------------------------------------------------------------

To unsubscribe, e-mail:

[email protected]For additional commands, e-mail:

[email protected]