[Hadoop-common-user] Moving Files to Distributed Cache in MapReduce

Slight modification: I now know how to add files to the distributed file
cache, which can be done via this command placed in the main or run class:

DistributedCache.addCacheFile(new URI("/user/hadoop/thefile.dat"),
conf);

However I am still having trouble locating the file in the distributed
cache. *How do I call the file path of thefile.dat in the distributed cache
as a string?* I am using Hadoop 0.20.2

Did you try using -files option in your hadoop jar command as:

/usr/bin/hadoop jar <jar name> <main class name> -files <absolute path of
file to be added to distributed cache> <input dir> <output dir>

After moving it to the distributed cache, how would I call it within my
MapReduce program?

I hope my previous reply helps...

Please unsubscribe me.

Thanks for the response! However, I'm having an issue with this line

Path[] cacheFiles = DistributedCache.getLocalCacheFiles(conf);

because conf has private access in org.apache.hadoop.configured

Is this what you are looking for?

http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

search for jobConf

jobConf is deprecated in 0.20.2 I believe; you're supposed to be using
Configuration for that

Hi all, I have now resolved my issue by doing a try/catch statement. Thanks
for all the help!

I could have sworn that I gave an example earlier this week on how to push and pull stuff from distributed cache.

Here's the meat of my post earlier...
Sample code on putting a file on the cache:
DistributedCache.addCacheFile(new URI(path+"MyFileName",conf));

Sample code in pulling data off the cache:
private Path[] localFiles = DistributedCache.getLocalCacheFiles(context.getConfiguration());
boolean exitProcess = false;
int i=0;
while (!exit){
fileName = localFiles[i].getName();
if (fileName.equalsIgnoreCase("model.txt")){
// Build your input file reader on localFiles[i].toString()
exitProcess = true;
}
i++;
}


Note that this is SAMPLE code. I didn't trap the exit condition if the file isn't there and you go beyond the size of the array localFiles[].
Also I set exit to false because its easier to read this as "Do this loop until the condition exitProcess is true".

When you build your file reader you need the full path, not just the file name. The path will vary when the job runs.

HTH

-Mike

We really need to build a working example to the wiki and add a link from the FAQ page. Any volunteers?

Yeah,

I'll write something up and post it on my web site. Definitely not InfoQ stuff, but a simple tip and tricks stuff.

-Mike

ok for accessing it in mapper code, u can do something like: