FAQ
Hi

We've run into problems w/ LockFactory usage on our system. The problem is
that the system can be such that the index is configured on a local file
system, or a remote, shared one. If remote/share, the protocol is sometimes
SMB 1.0/2.0, Samba, NFS 3/4 etc. In short, we have no control on the FS type
used or OS (Linux, Windows 2003/2008, Unix, AIX, iSeries etc.).

We currently use NativeFSLockFactory, because we want to ensure the lock
file is deleted upon JVM exit, whether by clean exit, or "kill". However,
the more I read the documentation of the class, and the inline documentation
in the class, I get a feeling that this choice, in our environment, is error
prone. Some file systems do not support native FS locking, some may take too
long to obtain the lock, in which case the Lock obtain method will timeout
etc. The current problem is that the index is configured on a Samba share,
and we hit LockObtainFailedExceptions (timeouts, or Access Denied once).
When the system is configured on a Windows share (I think SMB 1.0),
everything works fine.

So I'm not sure what to do. Should we always use SimpleFSLockFactory? We
know for sure only one node in the cluster will access the shared file
system for writing (not in a given time, but always). Therefore I was
tempted to move to NoLockFactory. But this move scares me a bit, and I'd
appreciate if someone can confirm my understanding of Lock being used to
protect against multiple IndexWriters from different JVMs *only*, or does it
have other uses as well? What about multiple IndexWriters from the same JVM?

If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
since we can guarantee by other means that only one IndexWriter attempts to
write to the index, we can IndexWriter.unlock(Dir) if the lock exists.

Also, the code of NativeFSLockFactory mentions that on NFS if not configured
properly, the lock obtaining may take 35 seconds. What sort of configuration
is needed? Just out of curiosity.

Any advice will be appreciated.

Shai

Search Discussions

  • Uwe Schindler at Dec 2, 2009 at 1:03 pm
    Hi Shai,
    have other uses as well? What about multiple IndexWriters from the same
    JVM?
    This also needs a lock. But if it is guaranteed to only have one JVM
    accessing the index, you can use an in-JVM-Locking mechanism which is
    provided by
    http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
    stanceLockFactory.html
    If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
    since we can guarantee by other means that only one IndexWriter attempts
    to
    write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
    If you can guarantee that you only have *one* JVM and there may be multiple
    IndexWriters, use above solution.

    Uwe


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Shai Erera at Dec 2, 2009 at 1:24 pm
    I have multiple JVMs on different machines accessing the shared file system.
    I don't really have multiple IndexWriters on the same JVM, I asked this just
    out of curiosity.

    So I don't understand from your reply if it's safe to use NoLockFactory, or
    I should use SimpleFSLockFactory and unlock if needed?

    Shai
    On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler wrote:

    Hi Shai,
    have other uses as well? What about multiple IndexWriters from the same
    JVM?
    This also needs a lock. But if it is guaranteed to only have one JVM
    accessing the index, you can use an in-JVM-Locking mechanism which is
    provided by

    http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
    stanceLockFactory.html<http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html>
    If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
    since we can guarantee by other means that only one IndexWriter attempts
    to
    write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
    If you can guarantee that you only have *one* JVM and there may be multiple
    IndexWriters, use above solution.

    Uwe


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Dec 2, 2009 at 1:46 pm
    Currently the sole usage of the LockFactory within Lucene is for the
    write.lock, ie, to prevent more than one writer (= IndexWriter, or,
    IndexReader trying to do deletions or set norms) from being open on
    the index at once.

    If your app has external mechanisms that can ensure, without fail,
    that only one writer will be open at once, then NoLockFactory is
    perfectly safe.

    But you have to be certain: one slipup will quickly corrupt the index.
    IndexWriters are like the male Siamese fighting fish (betta fish) --
    they will fight to the death [of the index]!

    Mike
    On Wed, Dec 2, 2009 at 8:24 AM, Shai Erera wrote:
    I have multiple JVMs on different machines accessing the shared file system.
    I don't really have multiple IndexWriters on the same JVM, I asked this just
    out of curiosity.

    So I don't understand from your reply if it's safe to use NoLockFactory, or
    I should use SimpleFSLockFactory and unlock if needed?

    Shai
    On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler wrote:

    Hi Shai,
    have other uses as well? What about multiple IndexWriters from the same
    JVM?
    This also needs a lock. But if it is guaranteed to only have one JVM
    accessing the index, you can use an in-JVM-Locking mechanism which is
    provided by

    http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
    stanceLockFactory.html<http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html>
    If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
    since we can guarantee by other means that only one IndexWriter attempts
    to
    write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
    If you can guarantee that you only have *one* JVM and there may be multiple
    IndexWriters, use above solution.

    Uwe


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Shai Erera at Dec 2, 2009 at 1:52 pm
    Thanks Mike.

    I'm pretty sure about our mechanism to control one JVM opening the index for
    write, but bugs can always be discovered. That's why we used
    NativeFSLockFactory - because even if the JVM is killed, the lock file
    should be removed. So if we do find the lock, it's fishy (Siamese or not).
    But if I use SimpleFSLockFactory and find the lock, it will be harder for me
    to tell if it's because of JVM kill, or because there is a bug and another
    IndexWriter opens the index for write.

    I think I'll go w/ NoLockFactory then ...

    Shai
    On Wed, Dec 2, 2009 at 3:45 PM, Michael McCandless wrote:

    Currently the sole usage of the LockFactory within Lucene is for the
    write.lock, ie, to prevent more than one writer (= IndexWriter, or,
    IndexReader trying to do deletions or set norms) from being open on
    the index at once.

    If your app has external mechanisms that can ensure, without fail,
    that only one writer will be open at once, then NoLockFactory is
    perfectly safe.

    But you have to be certain: one slipup will quickly corrupt the index.
    IndexWriters are like the male Siamese fighting fish (betta fish) --
    they will fight to the death [of the index]!

    Mike
    On Wed, Dec 2, 2009 at 8:24 AM, Shai Erera wrote:
    I have multiple JVMs on different machines accessing the shared file system.
    I don't really have multiple IndexWriters on the same JVM, I asked this just
    out of curiosity.

    So I don't understand from your reply if it's safe to use NoLockFactory, or
    I should use SimpleFSLockFactory and unlock if needed?

    Shai
    On Wed, Dec 2, 2009 at 3:03 PM, Uwe Schindler wrote:

    Hi Shai,
    have other uses as well? What about multiple IndexWriters from the
    same
    JVM?
    This also needs a lock. But if it is guaranteed to only have one JVM
    accessing the index, you can use an in-JVM-Locking mechanism which is
    provided by
    http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn
    stanceLockFactory.html<
    http://lucene.apache.org/java/3_0_0/api/all/org/apache/lucene/store/SingleIn%0AstanceLockFactory.html
    If NoLockFactory is not safe, then we can move to SimpleFSLockFactory, and
    since we can guarantee by other means that only one IndexWriter
    attempts
    to
    write to the index, we can IndexWriter.unlock(Dir) if the lock exists.
    If you can guarantee that you only have *one* JVM and there may be
    multiple
    IndexWriters, use above solution.

    Uwe


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedDec 2, '09 at 12:36p
activeDec 2, '09 at 1:52p
posts5
users3
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase