FAQ
0.20: Allow block reports to proceed without holding FSDataset lock
-------------------------------------------------------------------

Key: HDFS-2379
URL: https://issues.apache.org/jira/browse/HDFS-2379
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.20.206.0
Reporter: Todd Lipcon
Priority: Critical


As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.

This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Todd Lipcon (Resolved) (JIRA) at Nov 2, 2011 at 5:27 am
    [ https://issues.apache.org/jira/browse/HDFS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Todd Lipcon resolved HDFS-2379.
    -------------------------------

    Resolution: Fixed
    Fix Version/s: 0.20.206.0
    Hadoop Flags: Reviewed

    Committed to 0.20-security. Thanks for the reviews, Suresh.
    0.20: Allow block reports to proceed without holding FSDataset lock
    -------------------------------------------------------------------

    Key: HDFS-2379
    URL: https://issues.apache.org/jira/browse/HDFS-2379
    Project: Hadoop HDFS
    Issue Type: Bug
    Components: data-node
    Affects Versions: 0.20.206.0
    Reporter: Todd Lipcon
    Assignee: Todd Lipcon
    Priority: Critical
    Fix For: 0.20.206.0

    Attachments: hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt


    As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.
    This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Matt Foley (Reopened) (JIRA) at Feb 3, 2012 at 7:54 pm
    [ https://issues.apache.org/jira/browse/HDFS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Matt Foley reopened HDFS-2379:
    ------------------------------


    There's been a request to include this in 1.0.1, but the patch doesn't apply; it conflicts with another patch on several files.

    Todd, would you be able to look at making the patch work in branch-1.0, please?
    0.20: Allow block reports to proceed without holding FSDataset lock
    -------------------------------------------------------------------

    Key: HDFS-2379
    URL: https://issues.apache.org/jira/browse/HDFS-2379
    Project: Hadoop HDFS
    Issue Type: Bug
    Components: data-node
    Affects Versions: 1.1.0
    Reporter: Todd Lipcon
    Assignee: Todd Lipcon
    Priority: Critical
    Fix For: 1.1.0

    Attachments: hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt


    As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.
    This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Suresh Srinivas (Resolved) (JIRA) at Feb 4, 2012 at 2:18 am
    [ https://issues.apache.org/jira/browse/HDFS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Suresh Srinivas resolved HDFS-2379.
    -----------------------------------

    Resolution: Fixed
    Fix Version/s: 1.0.1

    Matt, I merged the changes from branch 1 to 1.0.
    0.20: Allow block reports to proceed without holding FSDataset lock
    -------------------------------------------------------------------

    Key: HDFS-2379
    URL: https://issues.apache.org/jira/browse/HDFS-2379
    Project: Hadoop HDFS
    Issue Type: Bug
    Components: data-node
    Affects Versions: 1.1.0
    Reporter: Todd Lipcon
    Assignee: Todd Lipcon
    Priority: Critical
    Fix For: 1.1.0, 1.0.1

    Attachments: hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt


    As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.
    This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedSep 28, '11 at 9:05a
activeFeb 4, '12 at 2:18a
posts4
users1
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase