[OpenRelevance-dev] some general questions

For Lucy, ideally I want us to be benchmarking relevancy just like we
benchmark performance. Two major features of Lucy are pluggable index
components and the ability to do rapid prototyping in the host language. We
should have some criteria by which we judge the crazy ideas people come up
with and we should be able to single out and reward the good ones.

One thing we don't do by default in Lucene is take proximity of term matches
into account. I'd like to measure how much relevance improves when we augment
basic boolean queries with span queries. That would make it easier to make a
sensible engineering tradeoff between responsiveness and relevance. And maybe
if the improvements are significant enough, somebody will build an optimized
scoring class tree which exploits that information.
Code-wise, changing up Similarity impelentations and settings is cake.

I still don't understand exactly how we judge ranking with these collections,
though. My understanding is that the collections mark each document as either
1 or 0, meaning "relevant or not relevant". That's fine for precision and
recall...

http://en.wikipedia.org/wiki/Precision_and_recall

... but how do we use these materials to judge relative ranking of successful
matches when the scores in the answer book are all exactly 1.0?
ORP should definitely do that, if for no other reason than that there seems to
be consensus that other search engines need to be supported.

If it makes sense for Lucene or Lucy to extend the ORP benchmark code, we can
choose to do so. If we can share at least some of the benchmarking framework
code across projects, that would be great. I'm not sure whether that will work
out well in practice, but it's worth a try for Lucy at least -- in a worst
case scenario, we fork and port.
I would kind of like to see Open Relevance become the project that sets the
standard for scientifically rigorous benchmarking, including but of course not
limited to competitive performance benchmarking.

There's some overlap between the tasks of benchmarking performance and
benchmarking relevance.
Each project will still need to implement its own client to interface with
the Open Relevance benchmarking API, but I think a lot can be shared.
Agreed. But we should move forward with the corpuses we've got today.

Marvin Humphrey

Marvin, thanks for your ideas... i gave some quick comments below.
I think one difference here though is that with performance, you can
microbenchmark, although its tricky as you know.
With relevance its not really safe to do that in my opinion...
But for "big" ideas, we definitely can.

I think in general the more test collections we have, the more accurate the
picture becomes...
We don't need a perfect collection immediately, maybe lots of imperfect ones
will do the trick for the time being?

yes, this is a really good idea to evaluate, although my gut instinct tells
me the gains will be language-dependent to some extent,
the same challenge faced in MT when related words can be very far apart in a
sentence in some langs, and it comes out goobly-gook because
some phrase-based MT doesn't use "enough" context for that language.

if span queries help, i think it would be even better to produce an idea of
what kind of 'span context' you need for some different languages/document
types for good results.

oh you mean more fine-grained measurement where the top 100 docs are all
relevant, but some more relevant than others?
I guess you are right, but I think in general using precision/recall is a
good indicator that things are being ranked correctly across the board?

I think when i say 'scoring' i am usually referring to algorithms that are
very different from what lucene provides via Similarity.
For example, lucene does not provide 'average document length' in Similarity
to support a lot of the more modern formulas like BM25

(side note, I have measured nice gains on Hindi and Persian by using BM25 w/
avg doc length calculated from norms, which you read in anyway)

yeah i hate the current format. I only wanted to get things off the ground
quickly since lucene's benchmark pkg can already deal with it.
And even lucene's "quality" pkg in the benchmark pkg is so limited:
for example it only runs topic queries but most of the time you want to also
be able to run topic+description or topic+description+narrative.
You also have to change code (at query time) to swap in a new analyzer, or
whatever... you can change analyzer in the .alg file at index time though.

I hadn't really thought about performance as well, but you are right, we
should keep this in mind. that's a good idea, its part of the whole picture.

ok, lets start thinking of some ways to improve this. currently you have to
apply ORP-2 to fix the README instructions,
but then anyone should be able to run the current stuff if they want, and
maybe have ideas for improvements.

Move forward doing more tests/producing worthwhile relevance experiments for
projects like Lucene?
Do you think this might create more developer interest for ORP,
or is the problem that everyone is already familiar with relevance testing,
but just has no time?

Another difference is that it's important to perform multiple iterations for
performance microbenchmarks, while the results of a relevance benchmark
should be the same every time.

Nevertheless, the rough outline is the same for search-time benchmarking.

* Acquire a collection of documents, either by parsing and optionally
manipulating an existing collection, or by creating documents artificially.
* Issue instructions to the client code to build the index according to some
plan.
* Launch a client process which runs the queries, storing the results to an
intermediate format such as JSON or XML.
* Retrieve and parse the client output.
* Perform statistical processing as required (e.g. find the truncated mean
for performance benchmarks).
* Generate a formatted report.
OK, I trust that existing academic approaches are sufficient for measuring and
refining relevance. If that's truly what's done, then all we need to do is
apply it.
Right, and this is where pluggable scoring components come in. "Flexible
Indexing" a la LUCENE-1458, too, provided that the eventual interface is
simple enough to be practical.
It's going to be a little tricky dividing up responsibilities between client
code and the core ORP benching tools. The line between data description and
instruction specification is fuzzy when describing algorithms.

I think we will want to leave more to the client rather than less so that
optimization is fully in the hands of the client. We'll have to trust that
the client implementation is correct.
Perhaps we could proceed like so:

1. Build a benchmarking framework which leverages existing collections.
2. Use some slice of the Apache email archives and some sort of
crowdsourcing to create a deeply flawed but still useful corpus.
3. Build a better corpus using lessons learned.

The problem is that creating a spiffy corpus is labor-intensive and thus
expensive. Everybody wants that, but it's hard to get over the initial hump.
So let's try to achieve some intermediate objectives first.

Marvin Humphrey

This is definitely a major component and in a lot of ways the reason this project was started. We need a way for us developers to be able to talk about relevance in a standard way using all open materials so that we can compare relevance. For instance, length norm is most likely favoring shorter documents in Lucene right now, what if we were to change it?

The thing about benchmark is it is pluggable, so making it possible to change Sim. should not be too hard.

Yeah, I think we will have to write some code. There was some talk originally about crowd-sourcing tools, too, for collecting judgments, etc.

Agreed.

It would be nice to use crowdsourcing for this. It does need to be on Mechanical Turk. Crowdsourcing is a paradigm for relevance evaluation that could be useful here.

o.

+1

Ooops. Thanks for the correction :).

o.

--- On Fri, 11/20/09, Grant Ingersoll wrote: