[Hadoop-common-user] Initialization costs

We have been doing some similar things but we use custom MapRunner
classes to load the resouces, for example files that need to be opened
or a shared cache to reduce network reads, once per Map split and then
pass the resources into the Map tasks. Here is an example of what it
might look like:

Dennis

public class YourRunner
implements MapRunnable {

private JobConf job;
private YourMapper mapper;
private Class inputKeyClass;
private Class inputValueClass;

public void configure(JobConf job) {
this.job = job;
this.inputKeyClass = job.getInputKeyClass();
this.inputValueClass = job.getInputValueClass();
}

private void closeReaders(MapFile.Reader[] readers) {

if (readers == null)
return;
for (int i = 0; i < readers.length; i++) {
try {
readers[i].close();
}
catch (Exception e) {

}
}
}

public void run(RecordReader input, OutputCollector output, Reporter
reporter)
throws IOException {

final FileSystem fs = FileSystem.get(job);

Configuration conf = NutchConfiguration.create();
mapper = new YourMapper();

Path filesPath= new Path(parent, mapfiledir);
MapFile.Reader[] readers= MapFileOutputFormat.getReaders(fs,
filesPath, conf);
Map <Integer, String> cache= new HashMap <Integer, String>();

mapper.setCache(cache);
mapper.setReaders(readers);

try {

WritableComparable key =
(WritableComparable)job.newInstance(inputKeyClass);
Writable value = (Writable)job.newInstance(inputValueClass);

while (input.next(key, value)) {
mapper.map(key, value, output, reporter);
}
}
finally {
mapper.close();
}

closeReaders(levelReaders);
}
}


Grant Ingersoll wrote:

A new JVM is used per task, so they'll need to be re-read per-task.

The job jar is cached locally by the tasktracker. So it is only copied
from dfs to the local disk once per job. Its content is shared by all
tasks in that job. So you can include shared files in a job jar. Tasks
are run connected to a directory with the unpacked jar contents.

You can also add other files that should be cached to your job, using
the DistributedCache API:

http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/filecache/DistributedCache.html

But including things in the job jar is far simpler.

Doug

Grant Ingersoll wrote: