FAQ
Dear Friend,
I have encountered some performance problems recently in lucene
search 2.9. I use a single IndexSearcher in the whole system, It seems
perfect when there is less than 10 threads doing search concurrenty.
Bu if there is more than 100 threads doing concurrent search,the
average response time is becoming bigger(>1s),and the max response
time reaches 299s. I really don't know how to improve,can you help me?
Thanks a lot !

Wilson
2009.10.23
The profiling result about 400 concurret search is at:
http://i3.6.cn/cvbnm/aa/f5/00/63521d982a469f5063b82268eee91d08.gif
it seems a lot of time consumed by TermScorer.score.
Follewing is my servlet class which is reponse to search request:
public final class DispatchServlet extends
javax.servlet.http.HttpServlet implements javax.servlet.Servlet
{
private static final long serialVersionUID = -5547647006004900451L;
protected final Log log = LogFactory.getLog(getClass());
protected Searcher searcher;
protected Directory dir;
protected RAMDirectory ram;

public DispatchServlet() {
super();
}

public void init() throws ServletException {
super.init();
try {

dir = FSDirectory.open(new
File("/usr/bestv/search_engin_index/index/program"));
ram = new RAMDirectory(dir);
searcher = new IndexSearcher(ram,true);
int h = searcher.search(tq,null,1).totalHits;
System.out.println("the searcher has warmed and
searched " + h + " docs" );
}
} catch (IOException e) {
log.error(e);
}
}

protected void doPost(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
response.setContentType("text/html");
doExecute(request.getParameter("q"),response);
}

protected void doGet(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
response.setContentType("text/html");
try{
String schCon =
URLDecoder.decode(request.getParameter("q"),"UTF-8");
doExecute(schCon,response);
}catch(Exception e){
response.getWriter().write("Parameter
Error,please send param 'q'");
}
}

public void doExecute(String schCon,HttpServletResponse response)
throws ServletException,IOException{
response.getWriter().write(new SearchCommand().search(searcher));
}

}

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Yonik Seeley at Oct 23, 2009 at 2:35 pm
    How many processors do you have on this system?
    If you are CPU bound, 100 threads is going to be 10 times slower (at a
    minimum) than 10 threads (unless you have more than 10 CPUs).

    -Yonik
    http://www.lucidimagination.com
    On Fri, Oct 23, 2009 at 2:18 AM, Wilson Wu wrote:
    Dear Friend,
    I have encountered some performance problems recently in lucene
    search 2.9. I use a single IndexSearcher in the whole system, It seems
    perfect when there is less than 10 threads doing search concurrenty.
    Bu if there is more than 100 threads doing concurrent search,the
    average response time is becoming bigger(>1s),and the max response
    time reaches 299s. I really don't know how to improve,can you help me?
    Thanks a lot !

    Wilson
    2009.10.23
    The profiling result about 400 concurret search is at:
    http://i3.6.cn/cvbnm/aa/f5/00/63521d982a469f5063b82268eee91d08.gif
    it seems a lot of time consumed by TermScorer.score.
    Follewing is my servlet class which is reponse to search request:
    public final class DispatchServlet extends
    javax.servlet.http.HttpServlet implements javax.servlet.Servlet
    {
    private static final long serialVersionUID = -5547647006004900451L;
    protected final Log log = LogFactory.getLog(getClass());
    protected Searcher searcher;
    protected Directory dir;
    protected RAMDirectory ram;

    public DispatchServlet() {
    super();
    }

    public void init() throws ServletException {
    super.init();
    try {

    dir = FSDirectory.open(new
    File("/usr/bestv/search_engin_index/index/program"));
    ram = new RAMDirectory(dir);
    searcher = new IndexSearcher(ram,true);
    int h = searcher.search(tq,null,1).totalHits;
    System.out.println("the searcher has warmed and
    searched " + h + " docs" );
    }
    } catch (IOException e) {
    log.error(e);
    }
    }

    protected void doPost(HttpServletRequest request, HttpServletResponse
    response) throws ServletException, IOException {
    response.setContentType("text/html");
    doExecute(request.getParameter("q"),response);
    }

    protected void doGet(HttpServletRequest request, HttpServletResponse
    response) throws ServletException, IOException {
    response.setContentType("text/html");
    try{
    String schCon =
    URLDecoder.decode(request.getParameter("q"),"UTF-8");
    doExecute(schCon,response);
    }catch(Exception e){
    response.getWriter().write("Parameter
    Error,please send param 'q'");
    }
    }

    public void doExecute(String schCon,HttpServletResponse response)
    throws ServletException,IOException{
    response.getWriter().write(new SearchCommand().search(searcher));
    }

    }

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Wilson Wu at Oct 24, 2009 at 1:50 pm
    Hi,
    Thanks a lot for your reply. There are 4 processors in my system.
    I am not sure that 100 threads is going to be 10 times slower
    than 10 threads .Because all the threads don't run serial but
    parallel. I think when there are 100 customers accessing my system,100
    http connections will be builded and all the http threads will be
    responsed by the Servlet at the same time. My question is :
    when 100 threads do search concurrently, are there any resources
    shared by differenet threads and these resources caused thread
    waiting/blocking in lucene IndexSearcher? If it dose, how can I avoid.
    Thanks again.

    wilson

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Wilson Wu at Oct 24, 2009 at 2:03 pm

    ---------- Forwarded message ----------
    From: Wilson Wu <songzi0206@gmail.com>
    Date: 2009/10/24
    Subject: Re: help needed improving lucene concurret search performance
    To: java-user@lucene.apache.org
    Hi,
    Thanks a lot for your reply. There are 4 processors in my system.
    I am not sure that 100 threads is going to be 10 times slower
    than 10 threads .Because all the threads don't run serial but
    parallel. I think when there are 100 customers accessing my system,100
    http connections will be builded and all the http threads will be
    responsed by the Servlet at the same time. My question is :
    when 100 threads do search concurrently, are there any resources
    shared by differenet threads and these resources caused thread
    waiting/blocking in lucene IndexSearcher? If it dose, how can I avoid.
    Thanks again.

    wilson

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedOct 23, '09 at 6:18a
activeOct 24, '09 at 2:03p
posts4
users2
websitelucene.apache.org

2 users in discussion

Wilson Wu: 3 posts Yonik Seeley: 1 post

People

Translate

site design / logo © 2022 Grokbase