FAQ
Hi,

I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

Please help me to understand my mistakes:

- When should I close the IndexWriter ? Each time I add/update a document or never ?
- Should my java commit/read methods be synchronized ?
- Should I lock the directory and what's the best way to do it ?

I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

I use lucene-2.4.1.

Thanks in advance,
Tom

Search Discussions

  • Michael McCandless at Jan 8, 2010 at 12:01 pm
    Normally, this (using an IndexReader, [re-]opening a new IndexReader
    while an IndexWriter is committing) is perfectly fine. The reader
    searches the point-in-time snapshot of the index as of when it was
    opened.

    But: what filesystem are you using? NFS presents challenges, for example.

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Legrand thomas at Jan 9, 2010 at 12:56 pm
    Michael,

    The exception only occurs when the writer commits. But but the IndexReader can keep on reading.
    The searches are performed by the IndexSearcher using the IndexReader.
    My filesystem is ext2fs.

    I give a few details below about the way I use them; the FNF exception occurs in the "commitIndexWriter" method.

    Thanks,
    Tom


    // both reader and writer and static
    protected static IndexWriter mAdIndexWriter=null;
    protected static IndexReader mAdIndexReader=null;
    // misc
    protected static IndexSearcher mAdIndexSearcher=null;
    protected static Analyzer mAdIndexAnalyser=null;


    // create the writer once
    private void initIndexWriter() throws EncheromaxException {
    Directory adIndexDir=null;

    // Where is the index ?
    String indexLocation=mConfigProperties.getProperty(PropertiesEnum.ADVERTISEMENT_INDEX_LOCATION);

    // Get the ad's analyser
    Analyzer analyser = getAdvertisementIndexAnalyser();

    // create the writer
    if(mAdIndexWriter==null){
    try{
    adIndexDir=FSDirectory.getDirectory(indexLocation);
    boolean isTheIndexNew=true;
    mAdIndexWriter = new IndexWriter(adIndexDir, analyser,isTheIndexNew,IndexWriter.MaxFieldLength. UNLIMITED) ;

    /* ... */
    }catch(Exception ex){
    /*...*/
    }
    }
    }


    // get the reader
    private IndexReader getAdvertisementIndexReader() throws EncheromaxException {
    String indexLocation=null;
    Directory adIndexDir=null;

    // Where is the index ?
    try{
    indexLocation=mConfigProperties.getProperty(PropertiesEnum.ADVERTISEMENT_INDEX_LOCATION);
    adIndexDir=FSDirectory.getDirectory(indexLocation);
    }catch(Exception direx){
    /*...*/
    }

    // create the reader if it doesn't exist yet
    if(mAdIndexReader==null){
    try{
    mAdIndexReader= IndexReader.open(adIndexDir);
    }catch(Exception ex){
    /*..*/
    }
    }else{
    try{
    /* do we need to re-open it ? */
    if(mAdIndexReader.isCurrent()==false){
    // synchronize on the reader
    synchronized(mAdIndexReader){
    mAdIndexReader= IndexReader.open(adIndexDir);
    }
    }
    }catch(Exception ioex){
    /*...*/
    throw new EncheromaxException(CommonErrorEnum.INDEX_READER_CREATION_ERROR);
    }
    }
    return mAdIndexReader;
    }


    // get the searcher when I want to search
    private IndexSearcher getAdvertisementIndexSearcher() throws EncheromaxException {
    // create the searcher if it doesn't exist yet
    try{
    if(mAdIndexSearcher==null){
    try{
    mAdIndexSearcher= new IndexSearcher(getAdvertisementIndexReader());

    }catch(Exception ex){
    /*...*/
    }
    }

    else if(!mAdIndexSearcher.getIndexReader().isCurrent()){
    synchronized(mAdIndexSearcher){
    mAdIndexSearcher=new IndexSearcher(getAdvertisementIndexReader());
    }
    }
    }catch(Exception ioex){
    /*...*/
    }
    return mAdIndexSearcher;
    }



    // whenever I need to commit
    public synchronized void commitIndexWriter() throws EncheromaxException{
    try{
    mAdIndexWriter.commit();

    }catch(Exception docex){
    /* THE FILE NOT FOUND EXCEPTION OCCURS HERE */
    }
    }


    // add a document
    private void addADocument(Document adDoc) throws EncheromaxException{
    try{
    mAdIndexWriter.addDocument(adDoc, getAdvertisementIndexAnalyser());
    commitIndexWriter();

    }catch(Exception docex){
    /*...*/
    }
    }


    --- En date de : Ven 8.1.10, Michael McCandless <lucene@mikemccandless.com> a écrit :

    De: Michael McCandless <lucene@mikemccandless.com>
    Objet: Re: Concurrent access IndexReader / IndexWriter - FileNotFoundException
    À: java-user@lucene.apache.org
    Date: Vendredi 8 Janvier 2010, 13h00

    Normally, this (using an IndexReader, [re-]opening a new IndexReader
    while an IndexWriter is committing) is perfectly fine.  The reader
    searches the point-in-time snapshot of the index as of when it was
    opened.

    But: what filesystem are you using?  NFS presents challenges, for example.

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Jan 9, 2010 at 1:51 pm
    Can you post the full FNFE stack trace?

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Legrand thomas at Jan 9, 2010 at 3:06 pm
    Here are two stack traces: add+remove a document:

    Tom

    ---

    Remove a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_0.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.removeAdvertisementFromIndex(AdvertisementManager.java:635)
    at com.encheromax.api.implementation.admin.ApiAdminBurnAdvertisement.executeRequest(ApiAdminBurnAdvertisement.java:210)
    at com.encheromax.controller.admin.Rule.doReqFinal(Rule.java:100)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

    Add a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_v.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.addAdvertisementDocument(AdvertisementManager.java:536)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.appendFullAdvertisementToIndex(AdvertisementManager.java:713)
    at com.encheromax.api.implementation.ApiFinalizeAdvertisement.executeRequest(ApiFinalizeAdvertisement.java:224)
    at com.encheromax.controller.advertisement.CreateAdvertisement.doReq(CreateAdvertisement.java:294)
    at net.tmp.fwk.fw1.controller.Security.doReqFinal(Security.java:135)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:619)






    --- En date de : Sam 9.1.10, Michael McCandless <lucene@mikemccandless.com> a écrit :

    De: Michael McCandless <lucene@mikemccandless.com>
    Objet: Re: Concurrent access IndexReader / IndexWriter - FileNotFoundException
    À: java-user@lucene.apache.org
    Date: Samedi 9 Janvier 2010, 14h51

    Can you post the full FNFE stack trace?

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Jan 9, 2010 at 5:54 pm
    Can you double check that you're not creating 2 writers on the same
    directory, somehow?

    Or: is there any other process that removes files from this directory?

    Answering your original questions...: commit/read does not require any
    external synchronization or locking. You should generally keep your
    IW open indefinitely and just periodically commit and/or get a new
    reader (IndexWriter.getReader()) as needed.

    Mike

    On Sat, Jan 9, 2010 at 10:06 AM, legrand thomas
    wrote:
    Here are two stack traces: add+remove a document:

    Tom

    ---

    Remove a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_0.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.removeAdvertisementFromIndex(AdvertisementManager.java:635)
    at com.encheromax.api.implementation.admin.ApiAdminBurnAdvertisement.executeRequest(ApiAdminBurnAdvertisement.java:210)
    at com.encheromax.controller.admin.Rule.doReqFinal(Rule.java:100)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

    Add a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_v.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.addAdvertisementDocument(AdvertisementManager.java:536)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.appendFullAdvertisementToIndex(AdvertisementManager.java:713)
    at com.encheromax.api.implementation.ApiFinalizeAdvertisement.executeRequest(ApiFinalizeAdvertisement.java:224)
    at com.encheromax.controller.advertisement.CreateAdvertisement.doReq(CreateAdvertisement.java:294)
    at net.tmp.fwk.fw1.controller.Security.doReqFinal(Security.java:135)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:619)






    --- En date de : Sam 9.1.10, Michael McCandless <lucene@mikemccandless.com> a écrit :

    De: Michael McCandless <lucene@mikemccandless.com>
    Objet: Re: Concurrent access IndexReader / IndexWriter -  FileNotFoundException
    À: java-user@lucene.apache.org
    Date: Samedi 9 Janvier 2010, 14h51

    Can you post the full FNFE stack trace?

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Legrand thomas at Jan 9, 2010 at 7:07 pm
    Ok Michael,

    Thank you for your answers, I'll check the numbers of writers as no other process uses the directory. I'll let you know the way I make it work.

    Tom

    --- En date de : Sam 9.1.10, Michael McCandless <lucene@mikemccandless.com> a écrit :

    De: Michael McCandless <lucene@mikemccandless.com>
    Objet: Re: Concurrent access IndexReader / IndexWriter - FileNotFoundException
    À: java-user@lucene.apache.org
    Date: Samedi 9 Janvier 2010, 18h54

    Can you double check that you're not creating 2 writers on the same
    directory, somehow?

    Or: is there any other process that removes files from this directory?

    Answering your original questions...: commit/read does not require any
    external synchronization or locking.  You should generally keep your
    IW open indefinitely and just periodically commit and/or get a new
    reader (IndexWriter.getReader()) as needed.

    Mike

    On Sat, Jan 9, 2010 at 10:06 AM, legrand thomas
    wrote:
    Here are two stack traces: add+remove a document:

    Tom

    ---

    Remove a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_0.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.removeAdvertisementFromIndex(AdvertisementManager.java:635)
    at com.encheromax.api.implementation.admin.ApiAdminBurnAdvertisement.executeRequest(ApiAdminBurnAdvertisement.java:210)
    at com.encheromax.controller.admin.Rule.doReqFinal(Rule.java:100)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)

    Add a document:

    java.io.FileNotFoundException: /home/ia/prod/current-deployment/indexes/advertisement/_v.cfs (No such file or directory)
    at java.io.RandomAccessFile.open(Native Method)
    at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
    at org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
    at org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
    at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
    at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
    at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:236)
    at org.apache.lucene.index.DocumentsWriter.applyDeletes(DocumentsWriter.java:915)
    at org.apache.lucene.index.IndexWriter.applyDeletes(IndexWriter.java:4339)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3579)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3450)
    at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3363)
    at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3408)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.commitIndexWriter(AdvertisementManager.java:1728)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.addAdvertisementDocument(AdvertisementManager.java:536)
    at com.encheromax.business.advertisement.implementation.AdvertisementManager.appendFullAdvertisementToIndex(AdvertisementManager.java:713)
    at com.encheromax.api.implementation.ApiFinalizeAdvertisement.executeRequest(ApiFinalizeAdvertisement.java:224)
    at com.encheromax.controller.advertisement.CreateAdvertisement.doReq(CreateAdvertisement.java:294)
    at net.tmp.fwk.fw1.controller.Security.doReqFinal(Security.java:135)
    at net.tmp.fwk.fw1.controller.Action.process(Action.java:302)
    at net.tmp.fwk.fw1.controller.Action.doPost(Action.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at com.encheromax.servlet.filter.EncheromaxBeanFilter.doFilter(EncheromaxBeanFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:619)






    --- En date de : Sam 9.1.10, Michael McCandless <lucene@mikemccandless.com> a écrit :

    De: Michael McCandless <lucene@mikemccandless.com>
    Objet: Re: Concurrent access IndexReader / IndexWriter -  FileNotFoundException
    À: java-user@lucene.apache.org
    Date: Samedi 9 Janvier 2010, 14h51

    Can you post the full FNFE stack trace?

    Mike
    On Fri, Jan 8, 2010 at 5:35 AM, legrand thomas wrote:
    Hi,

    I often get a FileNotFoundException when my single IndexWriter commits while the IndexReader also tries to read. My application is multithreaded (Tomcat uses the business APIs); I firstly thought the read/write access was thread-safe but I probably forget something.

    Please help me to understand my mistakes:

    - When should I close the IndexWriter ? Each time I add/update a document or never ?
    - Should my java commit/read methods be synchronized ?
    - Should I lock the directory and what's the best way to do it ?

    I referred to the Lucene FAQ ("why do I sometimes get a FileNotFoundException when I search and update my index at the same time?") but I did not disable any locking for processes searching or updating the index.

    I use lucene-2.4.1.

    Thanks in advance,
    Tom


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedJan 8, '10 at 10:35a
activeJan 9, '10 at 7:07p
posts7
users2
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase