FAQ
Hi Kim,

We didn't fix it in the end. I just ended up manually writing the
files to the cluster using the FileSystem class, and then reading them
back out again on the other side. Not terribly efficient as I guess
the point of DistributedCache is that the files get distributed to
every node, whereas I'm only writing to two or three nodes, then every
map-task is then trying to read back from those two or three nodes the
data are stored on.

Unfortunately I didn't have the will or inclination to investigate it
any further as I had some pretty tight deadlines to keep to and it
hasn't caused me any significant problems yet...

Thanks,

Jamie
On 5 October 2010 22:30, Kim Vogt wrote:
I'm experiencing the same problem.  I was hoping there were be a reply to
this.  Anyone? Bueller?

-Kim
On Fri, Jul 16, 2010 at 1:58 AM, Jamie Cockrill wrote:

Dear All,

We recently upgraded from CDH3b1 to b2 and ever since, all our
mapreduce jobs that use the DistributedCache have failed. Typically,
we add files to the cache prior to job startup, using
addCacheFile(URI, conf) and then get them on the other side, using
getLocalCacheFiles(conf). I believe the hadoop-core versions for these
are 0.20.2+228 and +320 respectively.

We then open the files and read them in using a standard FileReader,
using the toString on the path object as the constructor parameter,
which has worked fine up to now. However, we're now getting
FileNotFound exceptions when the file reader tries to open the file.

Unfortunately the cluster is on an airgapped network, but the
FileNotFound line comes out like:

java.io.FileNotFoundException:

/tmp/hadoop-hadoop/mapred/local/taskTracker/archive/master/path/to/my/file/filename.txt/filename.txt

Note, the duplication of filename.txt is deliberate. I'm not sure if
that's strange or not as this has previously worked absolutely fine.
Has anyone else experienced this? Apologies if this is known, I've
only just joined the list.

Many thanks,

Jamie

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 3 of 5 | next ›
Discussion Overview
groupcommon-user @
categorieshadoop
postedJul 16, '10 at 8:59a
activeOct 6, '10 at 9:04p
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase