FAQ
Hi all,

I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
have been working on developing a web tool for the project with a number of
collaborators and we are thinking of the system requirements now. The
requirements are listed as following:

1. The system is designed to used for evaluating the searching relevancy of
queries or search engines.
2. The system can be used on various commercial or open-source search
engines, such as Google, Bing, Lucene, Solr, etc.
3. The system should be able to accept and store test collections. The test
collections include a collection of queries and qrels.
4. The system should be able to run test collections and evaluate queries
or search engines with various evaluators. The system should provide with
certain standard evaluators, while user should be able to run customized
evaluators.
5. The system should be able to keep track of timestamps, queries, document
set, schema(including data model, analyzer, tokenizer, filter, etc.),
scoring model, evaluation metrics, searching results and relevancy score.

Any ideas? Thanks!

Sincerely,
Fan

Search Discussions

  • Grant Ingersoll at Jul 3, 2013 at 12:54 pm
    Hi Fan,
    On Jul 2, 2013, at 11:31 PM, Fan Jiang wrote:

    Hi all,

    I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
    have been working on developing a web tool for the project with a number of
    collaborators and we are thinking of the system requirements now. The
    requirements are listed as following:

    1. The system is designed to used for evaluating the searching relevancy of
    queries or search engines.
    I think we'll want to support various types of evaluations, both TREC-style as well as focus-group style evals.
    2. The system can be used on various commercial or open-source search
    engines, such as Google, Bing, Lucene, Solr, etc.
    Some evaluations won't make sense w/ some engines, but I like the idea that they engines are pluggable.
    3. The system should be able to accept and store test collections. The test
    collections include a collection of queries and qrels.
    4. The system should be able to run test collections and evaluate queries
    or search engines with various evaluators. The system should provide with
    certain standard evaluators, while user should be able to run customized
    evaluators.
    5. The system should be able to keep track of timestamps, queries, document
    set, schema(including data model, analyzer, tokenizer, filter, etc.),
    scoring model, evaluation metrics, searching results and relevancy score.

    Any ideas? Thanks!
    A few things come to mind for now:

    1. Simple, easy to use UI that others can easily pick up and extend
    2. Good documentation on using and extending

    Overall, I like this and glad to see someone picking back up on ORP.

    -Grant
  • Fan Jiang at Jul 6, 2013 at 8:25 pm
    Hi Grant,

    Thank you for your advices. I also come up with a new idea. The system
    should runs on several services - One for running evaluations and keeping
    track of the evaluation records, one for authentication and
    authorization(if the system is exposed to multi-users) and one works as a
    repository of test collections. In this way, the system can be more
    scalable and the business logic can be more clear. Does this make sense?
    Thanks!

    Sincerely,
    Fan


    2013/7/3 Grant Ingersoll <gsingers@apache.org>
    Hi Fan,
    On Jul 2, 2013, at 11:31 PM, Fan Jiang wrote:

    Hi all,

    I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
    have been working on developing a web tool for the project with a number of
    collaborators and we are thinking of the system requirements now. The
    requirements are listed as following:

    1. The system is designed to used for evaluating the searching relevancy of
    queries or search engines.
    I think we'll want to support various types of evaluations, both
    TREC-style as well as focus-group style evals.
    2. The system can be used on various commercial or open-source search
    engines, such as Google, Bing, Lucene, Solr, etc.
    Some evaluations won't make sense w/ some engines, but I like the idea
    that they engines are pluggable.
    3. The system should be able to accept and store test collections. The test
    collections include a collection of queries and qrels.
    4. The system should be able to run test collections and evaluate queries
    or search engines with various evaluators. The system should provide with
    certain standard evaluators, while user should be able to run customized
    evaluators.
    5. The system should be able to keep track of timestamps, queries, document
    set, schema(including data model, analyzer, tokenizer, filter, etc.),
    scoring model, evaluation metrics, searching results and relevancy score.

    Any ideas? Thanks!
    A few things come to mind for now:

    1. Simple, easy to use UI that others can easily pick up and extend
    2. Good documentation on using and extending

    Overall, I like this and glad to see someone picking back up on ORP.

    -Grant



    --
    Sincerely,
    Fan Jiang

    Master of Science in Information Science
    School of Library and Information Science
    University of North Carolina at Chapel Hill
    (919)-360-1047
  • Grant Ingersoll at Jul 8, 2013 at 1:34 am
    Yes, I think that makes sense, although I'd worry about authentication/authorization last, if possible.
    On Jul 6, 2013, at 4:24 PM, Fan Jiang wrote:

    Hi Grant,

    Thank you for your advices. I also come up with a new idea. The system
    should runs on several services - One for running evaluations and keeping
    track of the evaluation records, one for authentication and
    authorization(if the system is exposed to multi-users) and one works as a
    repository of test collections. In this way, the system can be more
    scalable and the business logic can be more clear. Does this make sense?
    Thanks!

    Sincerely,
    Fan


    2013/7/3 Grant Ingersoll <gsingers@apache.org>
    Hi Fan,
    On Jul 2, 2013, at 11:31 PM, Fan Jiang wrote:

    Hi all,

    I am Fan and I am new to Apache Lucene Open Relevance Project. Recently I
    have been working on developing a web tool for the project with a number of
    collaborators and we are thinking of the system requirements now. The
    requirements are listed as following:

    1. The system is designed to used for evaluating the searching relevancy of
    queries or search engines.
    I think we'll want to support various types of evaluations, both
    TREC-style as well as focus-group style evals.
    2. The system can be used on various commercial or open-source search
    engines, such as Google, Bing, Lucene, Solr, etc.
    Some evaluations won't make sense w/ some engines, but I like the idea
    that they engines are pluggable.
    3. The system should be able to accept and store test collections. The test
    collections include a collection of queries and qrels.
    4. The system should be able to run test collections and evaluate queries
    or search engines with various evaluators. The system should provide with
    certain standard evaluators, while user should be able to run customized
    evaluators.
    5. The system should be able to keep track of timestamps, queries, document
    set, schema(including data model, analyzer, tokenizer, filter, etc.),
    scoring model, evaluation metrics, searching results and relevancy score.

    Any ideas? Thanks!
    A few things come to mind for now:

    1. Simple, easy to use UI that others can easily pick up and extend
    2. Good documentation on using and extending

    Overall, I like this and glad to see someone picking back up on ORP.

    -Grant



    --
    Sincerely,
    Fan Jiang

    Master of Science in Information Science
    School of Library and Information Science
    University of North Carolina at Chapel Hill
    (919)-360-1047
    --------------------------------------------
    Grant Ingersoll | @gsingers
    http://www.lucidworks.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupopenrelevance-dev @
categorieslucene
postedJul 3, '13 at 3:31a
activeJul 8, '13 at 1:34a
posts4
users2
websitelucene.apache.org...

2 users in discussion

Grant Ingersoll: 2 posts Fan Jiang: 2 posts

People

Translate

site design / logo © 2018 Grokbase