Grokbase Groups Lucene dev June 2011
FAQ
Make FSDirectory.fsync() public and static
------------------------------------------

Key: LUCENE-3230
URL: https://issues.apache.org/jira/browse/LUCENE-3230
Project: Lucene - Java
Issue Type: New Feature
Components: core/store
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.3, 4.0


I find FSDirectory.fsync() (today protected and instance method) very useful as a utility to sync() files. I'd like create a FSDirectory.sync() utility which contains the exact same impl of FSDir.fsync(), and have the latter call it. We can have it part of IOUtils too, as it's a completely standalone utility.

I would get rid of FSDir.fsync() if it wasn't protected (as if encouraging people to override it). I doubt anyone really overrides it (our core Directories don't).

Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?

The patch is trivial, so I'd like to get some feedback before I post it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Search Discussions

  • Michael McCandless (JIRA) at Jun 23, 2011 at 2:12 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053872#comment-13053872 ]

    Michael McCandless commented on LUCENE-3230:
    --------------------------------------------

    This seems OK, but my only worry is.... I'm not sure this way of fsync'ing really "works"? Ie, this code opens a r/w RAF, calls sync, closes it. It's not clear that this is guaranteed to sync file handles open in the past against the same file. This is something we separately should look into / fix, but with this uncertainty it makes me nervous exposing this as a public API... maybe we could expose it with a big warning.

    bq. Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?

    +1
    Make FSDirectory.fsync() public and static
    ------------------------------------------

    Key: LUCENE-3230
    URL: https://issues.apache.org/jira/browse/LUCENE-3230
    Project: Lucene - Java
    Issue Type: New Feature
    Components: core/store
    Reporter: Shai Erera
    Assignee: Shai Erera
    Priority: Minor
    Fix For: 3.3, 4.0


    I find FSDirectory.fsync() (today protected and instance method) very useful as a utility to sync() files. I'd like create a FSDirectory.sync() utility which contains the exact same impl of FSDir.fsync(), and have the latter call it. We can have it part of IOUtils too, as it's a completely standalone utility.
    I would get rid of FSDir.fsync() if it wasn't protected (as if encouraging people to override it). I doubt anyone really overrides it (our core Directories don't).
    Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?
    The patch is trivial, so I'd like to get some feedback before I post it.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Shai Erera (JIRA) at Jun 24, 2011 at 4:39 am
    [ https://issues.apache.org/jira/browse/LUCENE-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054234#comment-13054234 ]

    Shai Erera commented on LUCENE-3230:
    ------------------------------------

    I opened 2 RAF instances over the same file and called getFD() on each. I've received 2 instances of FD, each had a different 'handle' value. So I agree it's not clear whether syncing one FD affects the other.

    I also found some posts on the web claiming that FS.sync() doesn't really work on all OS, and that some hardware manufacturers enable "hardware write caching", so even if the OS obeys the sync() call, the HW may not. I guess there's not much we can do about the HW case.
    * http://hardware.slashdot.org/story/05/05/13/0529252/Your-Hard-Drive-Lies-to-You
    * http://lists.apple.com/archives/darwin-dev/2005/Feb/msg00072.html (bad fsync on Mac OS X)

    So perhaps we shouldn't make it a public API after all. If someone can sync() on the same OutputStream he used to flush/close, it's better than how we do it today. I hate to introduce public API w/ big warnings.

    As for our case (Lucene usage of sync()), it'd be good if we can sync() in the IndexOutput that wrote the data. So maybe we should add sync() to IO? Not sure how it will play out in Directory, which today syncs file names, and not IndexOutput instances.

    Shall I close this issue then? Or rename it to handle the IO.sync() issue?
    Make FSDirectory.fsync() public and static
    ------------------------------------------

    Key: LUCENE-3230
    URL: https://issues.apache.org/jira/browse/LUCENE-3230
    Project: Lucene - Java
    Issue Type: New Feature
    Components: core/store
    Reporter: Shai Erera
    Assignee: Shai Erera
    Priority: Minor
    Fix For: 3.3, 4.0


    I find FSDirectory.fsync() (today protected and instance method) very useful as a utility to sync() files. I'd like create a FSDirectory.sync() utility which contains the exact same impl of FSDir.fsync(), and have the latter call it. We can have it part of IOUtils too, as it's a completely standalone utility.
    I would get rid of FSDir.fsync() if it wasn't protected (as if encouraging people to override it). I doubt anyone really overrides it (our core Directories don't).
    Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?
    The patch is trivial, so I'd like to get some feedback before I post it.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at Jun 24, 2011 at 10:37 am
    [ https://issues.apache.org/jira/browse/LUCENE-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054348#comment-13054348 ]

    Michael McCandless commented on LUCENE-3230:
    --------------------------------------------

    OK, I think close this issue, and open another (to switch to syncing the actual IOs we opened)... this will be a challenge for Lucene though.
    Make FSDirectory.fsync() public and static
    ------------------------------------------

    Key: LUCENE-3230
    URL: https://issues.apache.org/jira/browse/LUCENE-3230
    Project: Lucene - Java
    Issue Type: New Feature
    Components: core/store
    Reporter: Shai Erera
    Assignee: Shai Erera
    Priority: Minor
    Fix For: 3.3, 4.0


    I find FSDirectory.fsync() (today protected and instance method) very useful as a utility to sync() files. I'd like create a FSDirectory.sync() utility which contains the exact same impl of FSDir.fsync(), and have the latter call it. We can have it part of IOUtils too, as it's a completely standalone utility.
    I would get rid of FSDir.fsync() if it wasn't protected (as if encouraging people to override it). I doubt anyone really overrides it (our core Directories don't).
    Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?
    The patch is trivial, so I'd like to get some feedback before I post it.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Shai Erera (JIRA) at Jun 24, 2011 at 11:19 am
    [ https://issues.apache.org/jira/browse/LUCENE-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shai Erera resolved LUCENE-3230.
    --------------------------------

    Resolution: Won't Fix

    I opened LUCENE-3237 to improve how fsync works. After we move sync() to IndexOutput, a public static sync() API won't make much sense.
    Make FSDirectory.fsync() public and static
    ------------------------------------------

    Key: LUCENE-3230
    URL: https://issues.apache.org/jira/browse/LUCENE-3230
    Project: Lucene - Java
    Issue Type: New Feature
    Components: core/store
    Reporter: Shai Erera
    Assignee: Shai Erera
    Priority: Minor
    Fix For: 3.3, 4.0


    I find FSDirectory.fsync() (today protected and instance method) very useful as a utility to sync() files. I'd like create a FSDirectory.sync() utility which contains the exact same impl of FSDir.fsync(), and have the latter call it. We can have it part of IOUtils too, as it's a completely standalone utility.
    I would get rid of FSDir.fsync() if it wasn't protected (as if encouraging people to override it). I doubt anyone really overrides it (our core Directories don't).
    Also, while reviewing the code, I noticed that if IOE occurs, the code sleeps for 5 msec. If an InterruptedException occurs then, it immediately throws ThreadIE, completely ignoring the fact that it slept due to IOE. Shouldn't we at least pass IOE.getMessage() on ThreadIE?
    The patch is trivial, so I'd like to get some feedback before I post it.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira



    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieslucene
postedJun 22, '11 at 2:13p
activeJun 24, '11 at 11:19a
posts5
users1
websitelucene.apache.org

1 user in discussion

Shai Erera (JIRA): 5 posts

People

Translate

site design / logo © 2021 Grokbase