Hi people,
I've a cluster where around 40% of nodes are low on disk space. The output after the maps is too big for the mapred.local.dir in the nodes low on disk space and with hadoop it happens a lot that it tries to flush the output to these nodes, fails, tries on some other node until it finally flushes the data into one of the nodes with large disk space.
So my question is whether its possible for hadoop to select or for us to be able to notify hadoop of the nodes which have larger disk space so that it doesn't waste time on nodes with low disk space.
Many thanks
H
