Hi Kim,

We didn't fix it in the end. I just ended up manually writing the
files to the cluster using the FileSystem class, and then reading them
back out again on the other side. Not terribly efficient as I guess
the point of DistributedCache is that the files get distributed to
every node, whereas I'm only writing to two or three nodes, then every
map-task is then trying to read back from those two or three nodes the
data are stored on.

Unfortunately I didn't have the will or inclination to investigate it
any further as I had some pretty tight deadlines to keep to and it
hasn't caused me any significant problems yet...


On 5 October 2010 22:30, Kim Vogt wrote:
I'm experiencing the same problem.  I was hoping there were be a reply to
this.  Anyone? Bueller?

On Fri, Jul 16, 2010 at 1:58 AM, Jamie Cockrill wrote:

Dear All,

We recently upgraded from CDH3b1 to b2 and ever since, all our
mapreduce jobs that use the DistributedCache have failed. Typically,
we add files to the cache prior to job startup, using
addCacheFile(URI, conf) and then get them on the other side, using
getLocalCacheFiles(conf). I believe the hadoop-core versions for these
are 0.20.2+228 and +320 respectively.

We then open the files and read them in using a standard FileReader,
using the toString on the path object as the constructor parameter,
which has worked fine up to now. However, we're now getting
FileNotFound exceptions when the file reader tries to open the file.

Unfortunately the cluster is on an airgapped network, but the
FileNotFound line comes out like:



Note, the duplication of filename.txt is deliberate. I'm not sure if
that's strange or not as this has previously worked absolutely fine.
Has anyone else experienced this? Apologies if this is known, I've
only just joined the list.

Many thanks,


Search Discussions

Discussion Posts


Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 5 | next ›
Discussion Overview
groupcommon-user @
postedJul 16, '10 at 8:59a
activeOct 6, '10 at 9:04p



site design / logo © 2021 Grokbase