FAQ
Race condition creating S3 buffer directory for NativeS3FileSystem
------------------------------------------------------------------

Key: HADOOP-4377
URL: https://issues.apache.org/jira/browse/HADOOP-4377
Project: Hadoop Core
Issue Type: Bug
Components: fs/s3
Affects Versions: 0.18.1
Reporter: David Phillips


The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • David Phillips (JIRA) at Oct 8, 2008 at 3:48 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Attachment: hadoop-s3n-tmpdir.patch
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Oct 8, 2008 at 3:50 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Status: Patch Available (was: Open)
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Oct 16, 2008 at 8:13 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Attachment: hadoop-s3n-tmpdir.patch
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Oct 16, 2008 at 8:15 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Attachment: (was: hadoop-s3n-tmpdir.patch)
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Nov 7, 2008 at 2:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645675#action_12645675 ]

    Chris Douglas commented on HADOOP-4377:
    ---------------------------------------

    {noformat}
    - if (!dir.exists() && !dir.mkdirs()) {
    + if (!dir.exists() && !dir.mkdirs() && !dir.exists()) {
    {noformat}

    Isn't {{!dir.mkdirs() && !dir.exists()}} equivalent?
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Nov 7, 2008 at 2:53 am
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas reassigned HADOOP-4377:
    -------------------------------------

    Assignee: David Phillips
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Nov 7, 2008 at 5:37 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Attachment: hadoop-s3n-tmpdir.patch

    Chris, you are right, they should be equivalent. I verified that they both only do a single stat() if the directory exists. Patch updated.
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Nov 7, 2008 at 5:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Status: Open (was: Patch Available)
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Nov 7, 2008 at 5:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Release Note: Fix "Cannot create S3 buffer directory" race condition error
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • David Phillips (JIRA) at Nov 7, 2008 at 5:51 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    David Phillips updated HADOOP-4377:
    -----------------------------------

    Status: Patch Available (was: Open)
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hadoop QA (JIRA) at Nov 7, 2008 at 11:14 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645902#action_12645902 ]

    Hadoop QA commented on HADOOP-4377:
    -----------------------------------

    -1 overall. Here are the results of testing the latest attachment
    http://issues.apache.org/jira/secure/attachment/12393516/hadoop-s3n-tmpdir.patch
    against trunk revision 712247.

    +1 @author. The patch does not contain any @author tags.

    -1 tests included. The patch doesn't appear to include any new or modified tests.
    Please justify why no tests are needed for this patch.

    +1 javadoc. The javadoc tool did not generate any warning messages.

    +1 javac. The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs. The patch does not introduce any new Findbugs warnings.

    +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

    +1 core tests. The patch passed core unit tests.

    +1 contrib tests. The patch passed contrib unit tests.

    Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3551/testReport/
    Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3551/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
    Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3551/artifact/trunk/build/test/checkstyle-errors.html
    Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3551/console

    This message is automatically generated.
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Chris Douglas (JIRA) at Nov 7, 2008 at 11:46 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Chris Douglas updated HADOOP-4377:
    ----------------------------------

    Resolution: Fixed
    Fix Version/s: 0.20.0
    Hadoop Flags: [Reviewed]
    Status: Resolved (was: Patch Available)

    I just committed this. Thanks, David
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Fix For: 0.20.0

    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Hudson (JIRA) at Nov 8, 2008 at 4:49 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646000#action_12646000 ]

    Hudson commented on HADOOP-4377:
    --------------------------------

    Integrated in Hadoop-trunk #655 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/])
    . Fix a race condition in directory creation in
    NativeS3FileSystem. Contributed by David Phillips.

    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Fix For: 0.20.0

    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Robert Chansler (JIRA) at Mar 3, 2009 at 6:59 pm
    [ https://issues.apache.org/jira/browse/HADOOP-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Robert Chansler updated HADOOP-4377:
    ------------------------------------

    Release Note: (was: Fix "Cannot create S3 buffer directory" race condition error)

    No published release note for "just a bug."
    Race condition creating S3 buffer directory for NativeS3FileSystem
    ------------------------------------------------------------------

    Key: HADOOP-4377
    URL: https://issues.apache.org/jira/browse/HADOOP-4377
    Project: Hadoop Core
    Issue Type: Bug
    Components: fs/s3
    Affects Versions: 0.18.1
    Reporter: David Phillips
    Assignee: David Phillips
    Fix For: 0.20.0

    Attachments: hadoop-s3n-tmpdir.patch, hadoop-s3n-tmpdir.patch


    The buffer directory is checked for existence, then created if it doesn't exist. But the create can fail if the another process creates it in between. We can fix this by checking for existence again if the create fails. I've seen "Cannot create S3 buffer directory" occur, and this race is the most plausible explanation.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedOct 8, '08 at 3:44p
activeMar 3, '09 at 6:59p
posts15
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Robert Chansler (JIRA): 15 posts

People

Translate

site design / logo © 2022 Grokbase