FAQ
Changes to balancer bandwidth should not require datanode restart.
------------------------------------------------------------------

Key: HDFS-2171
URL: https://issues.apache.org/jira/browse/HDFS-2171
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer, data-node
Affects Versions: 0.20.205.0
Reporter: Eric Payne
Assignee: Eric Payne
Fix For: 0.20.205.0


Currently in order to change the value of the balancer bandwidth (dfs.datanode.balance.bandwidthPerSec), the datanode daemon must be restarted.

The optimal value of the bandwidthPerSec parameter is not always (almost never) known at the time of cluster startup, but only once a new node is placed in the cluster and balancing is begun. If the balancing is taking too long (bandwidthPerSec is too low) or the balancing is taking up too much bandwidth (bandwidthPerSec is too high), the cluster must go into a "maintenance window" where it is unusable while all of the datanodes are bounced. In large clusters of thousands of nodes, this can be a real maintenance problem because these "mainenance windows" can take a long time and there may have to be several of them while the bandwidthPerSec is experimented with and tuned.

A possible solution to this problem would be to add a -bandwidth parameter to the balancer tool. If bandwidth is supplied, pass the value to the datanodes via the OP_REPLACE_BLOCK and OP_COPY_BLOCK DataTransferProtocol requests. This would make it necessary, however, to change the DataTransferProtocol version.





--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Eric Payne (JIRA) at Jul 28, 2011 at 6:05 pm
    [ https://issues.apache.org/jira/browse/HDFS-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Eric Payne resolved HDFS-2171.
    ------------------------------

    Resolution: Duplicate

    Putting patches for both 0.20.205.0 and 0.23.0 on HDFS-2202.
    Changes to balancer bandwidth should not require datanode restart.
    ------------------------------------------------------------------

    Key: HDFS-2171
    URL: https://issues.apache.org/jira/browse/HDFS-2171
    Project: Hadoop HDFS
    Issue Type: Bug
    Components: balancer, data-node
    Affects Versions: 0.20.205.0, 0.23.0
    Reporter: Eric Payne
    Assignee: Eric Payne
    Fix For: 0.20.205.0, 0.23.0

    Attachments: HDFS-2171.patch


    Currently in order to change the value of the balancer bandwidth (dfs.datanode.balance.bandwidthPerSec), the datanode daemon must be restarted.
    The optimal value of the bandwidthPerSec parameter is not always (almost never) known at the time of cluster startup, but only once a new node is placed in the cluster and balancing is begun. If the balancing is taking too long (bandwidthPerSec is too low) or the balancing is taking up too much bandwidth (bandwidthPerSec is too high), the cluster must go into a "maintenance window" where it is unusable while all of the datanodes are bounced. In large clusters of thousands of nodes, this can be a real maintenance problem because these "mainenance windows" can take a long time and there may have to be several of them while the bandwidthPerSec is experimented with and tuned.
    A possible solution to this problem would be to add a -bandwidth parameter to the balancer tool. If bandwidth is supplied, pass the value to the datanodes via the OP_REPLACE_BLOCK and OP_COPY_BLOCK DataTransferProtocol requests. This would make it necessary, however, to change the DataTransferProtocol version.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
categorieshadoop
postedJul 19, '11 at 10:17p
activeJul 28, '11 at 6:05p
posts2
users1
websitehadoop.apache.org...
irc#hadoop

1 user in discussion

Eric Payne (JIRA): 2 posts

People

Translate

site design / logo © 2022 Grokbase