Ghost nodes in excluded node list for block allocation limit replication target count

Key: HDFS-1168
URL: https://issues.apache.org/jira/browse/HDFS-1168
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client, name-node
Reporter: Todd Lipcon

In HDFS-630 we added an excludedNodes parameter when allocating a block. In the case of a cluster that uses transient IPC ports, this list can accumulate past incarnations of restarted datanodes. Then, in NetworkTopology.countNumOfAvailableNodes, we count each of these "ghost" nodes against the total number of available nodes, and decide that there are no spots to place replicas, even though plenty are alive.

To reproduce, write data into HDFS with a very small block size (say 4KB) and then repeatedly kill and restart the local DN configured to use a transient port. After you have done so N times, where N is the number of nodes in the cluster, the NN will fail to allocate any targets even though N other nodes are still alive.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Search Discussions

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-dev @
postedMay 20, '10 at 7:00a
activeMay 20, '10 at 7:00a

1 user in discussion

Todd Lipcon (JIRA): 1 post



site design / logo © 2023 Grokbase