-------------------------------------------------------------------------------------
Key: HDFS-1168
URL: https://issues.apache.org/jira/browse/HDFS-1168
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client, name-node
Reporter: Todd Lipcon
In HDFS-630 we added an excludedNodes parameter when allocating a block. In the case of a cluster that uses transient IPC ports, this list can accumulate past incarnations of restarted datanodes. Then, in NetworkTopology.countNumOfAvailableNodes, we count each of these "ghost" nodes against the total number of available nodes, and decide that there are no spots to place replicas, even though plenty are alive.
To reproduce, write data into HDFS with a very small block size (say 4KB) and then repeatedly kill and restart the local DN configured to use a transient port. After you have done so N times, where N is the number of nodes in the cluster, the NN will fail to allocate any targets even though N other nodes are still alive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.