FAQ
trying to read configuration in hdfs write path might result in deadlock
------------------------------------------------------------------------

Key: HDFS-2081
URL: https://issues.apache.org/jira/browse/HDFS-2081
Project: Hadoop HDFS
Issue Type: Bug
Environment: MR-279 branch. This is specific to yarn since yarn tries to write the configuration file to hdfs while holding the lock of the configuration file.
Reporter: John George
Assignee: John George


While trying to run yarn with 1024 block size, it looks as though the dfsclients are hung waiting to write to DN. The reason is because yarn golds the configuration file lock and tries to write to hdfs which in turn tries to read from the configuration which requires the same lock.

The temporary fix is to make sure that dfsclient threads dont read the config in its write path. There might be better long term solutions like read/write locks and this needs to be investigated in another JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • John George (JIRA) at Jun 19, 2011 at 3:59 am
    [ https://issues.apache.org/jira/browse/HDFS-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    John George resolved HDFS-2081.
    -------------------------------

    Resolution: Duplicate

    Dupe of HADOOP-7082.
    trying to read configuration in hdfs write path might result in deadlock
    ------------------------------------------------------------------------

    Key: HDFS-2081
    URL: https://issues.apache.org/jira/browse/HDFS-2081
    Project: Hadoop HDFS
    Issue Type: Bug
    Affects Versions: 0.23.0
    Environment: MR-279 branch. This is specific to yarn since yarn tries to write the configuration file to hdfs while holding the lock of the configuration file.
    Reporter: John George
    Assignee: John George
    Fix For: 0.23.0

    Attachments: HDFS-2081.patch


    While trying to run yarn with 1024 block size, it looks as though the dfsclients are hung waiting to write to DN. The reason is because yarn holds the configuration file lock and tries to write to hdfs which in turn tries to read from the configuration which requires the same lock.
    The temporary fix is to make sure that dfsclient threads dont read the config in its write path. There might be better long term solutions like read/write locks and this needs to be investigated in another JIRA.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedJun 17, '11 at 10:55p
activeJun 19, '11 at 3:59a
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

John George (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase