FAQ
I am exploring the possibility of creating large lucene indices via ec2/s3.
Till now I have got only teh following url :
http://www.kimchy.org/lucene-and-amazon-s3/

But still dont know whether the lucene locking problem ( on a distributed FS
like S3/DFS ) is fixed or not. Any information is greatly apreciated.
--
View this message in context: http://www.nabble.com/lucene-on-amazon-s3-tp25254673p25254673.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Search Discussions

  • Simon Willnauer at Sep 2, 2009 at 10:16 am
    Hey there, AFAIK this problem on S3 has not been solved but anyway
    there might be other solutions to overcome this problem. As you are
    running on amazon anyway you might wanna consider to have some locking
    service like ZooKeeper (http://hadoop.apache.org/zookeeper/) which
    could help you with other problems in the future too. You might wanna
    look at the recipes section
    (http://hadoop.apache.org/zookeeper/docs/r3.2.0/recipes.html#sc_recipes_Locks)
    and create you distributed lock yourself.
    One more thing, I guess this is not a lucene-locking problem as the
    locking API is highly customizable see LockFactory
    (http://lucene.apache.org/java/2_4_0/api/core/org/apache/lucene/store/LockFactory.html)
    and Directory@setLockFactory
    (http://lucene.apache.org/java/2_4_0/api/core/org/apache/lucene/store/Directory.html#setLockFactory%28org.apache.lucene.store.LockFactory%29)

    simon

    On Wed, Sep 2, 2009 at 11:55 AM, prasenjitwrote:
    I am exploring the possibility of creating large lucene indices via ec2/s3.
    Till now I have got only teh following url :
    http://www.kimchy.org/lucene-and-amazon-s3/

    But still dont know whether the lucene locking problem ( on a distributed FS
    like S3/DFS ) is fixed or not. Any information is greatly apreciated.
    --
    View this message in context: http://www.nabble.com/lucene-on-amazon-s3-tp25254673p25254673.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
  • Michael McCandless at Sep 2, 2009 at 10:58 am
    So long as you can ensure, external to Lucene, that only one
    IndexWriter is open at once on the index, you can disable all of
    Lucene's normal locking. But you must be certain: if you accidentally
    allow two IndexWriter's to be open at once, it will quickly corrupt
    the index.

    Beyond locking, Lucene also generally requires client side cache
    coherence of the filesystem. Notably, NFS and the 2.0 version of
    Microsoft's SMB/CIFS (the default in Windows Server 7 & Vista), have
    cache coherence problems (SMB/CIFS 1.0 works fine). EG, if a writer
    somewhere updates the index, when a reader somewhere else lists the
    directory, is it guaranteed that the directory listing is always
    correct?

    If there are cache coherence problems, it usually manifests as a
    "false" FileNotFoundException... the good news is, you can catch this,
    wait a bit, and retry, as a workaround.

    If you get any data on S3's cache coherence, please post back!

    Mike

    On Wed, Sep 2, 2009 at 5:55 AM, prasenjitwrote:
    I am exploring the possibility of creating large lucene indices via ec2/s3.
    Till now I have got only teh following url :
    http://www.kimchy.org/lucene-and-amazon-s3/

    But still dont know whether the lucene locking problem ( on a distributed FS
    like S3/DFS ) is fixed or not. Any information is greatly apreciated.
    --
    View this message in context: http://www.nabble.com/lucene-on-amazon-s3-tp25254673p25254673.html
    Sent from the Lucene - Java Users mailing list archive at Nabble.com.


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
    For additional commands, e-mail: java-user-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupjava-user @
categorieslucene
postedSep 2, '09 at 9:56a
activeSep 2, '09 at 10:58a
posts3
users3
websitelucene.apache.org

People

Translate

site design / logo © 2022 Grokbase