Hello Hadoop Users list:
We are running Hadoop version 0.18.2. My team lead has asked me to investigate the answer to a particular question regarding Hadoop's handling of offline DataNodes - specifically, we would like to know how long a node can be offline before it is totally rebuilt when it has been readded to the cluster.
From what I've been able to determine from the documentation it appears to me that the NameNode will simply begin scheduling block replication on its remaining cluster members. If the offline node comes back online, and it reports all its blocks as being uncorrupted, then the NameNode just cleans up the "extra" blocks.
In other words, there is no explicit handling based on the length of the outage - the behavior of the cluster will depend entirely on the outage duration.
Anyone care to shed some light on this?