FAQ
Hi all,

This issue could very well be related to the Cloudera distribution
(CDH3b4) I use, but maybe someone knows the solution:

I configured a Job, something like this:

Configuration conf = getConf();
// ... set configuration
conf.set("mapred.jar", localJarFile.toString())
// tracker, zookeeper, hbase etc.


Job job = new Job(conf);
// map:
job.setMapperClass(DataImportMap.class);
job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(Put.class);
// reduce:

TableMapReduceUtil.initTableReducerJob("MyTable",
DataImportReduce.class, job);
FileInputFormat.addInputPath(job, new Path(inputData));

// execute:
job.waitForCompletion(true);

Now the server throws a strange exception below, see the stacktrace
below.

When i take look at the hdfs file system - through hdfs fuse - the file
is there, it really is the jar that contains my mapred classes.

Any clue wat goes wrong here?

Thanks,
Job


---------------------------------------------
java.io.FileNotFoundException:
File /var/lib/hadoop-0.20/cache/mapred/mapred/staging/job/.staging/job_201102241026_0002/job.jar does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
at
org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61)
at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
at
org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
at
org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at
org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212)
at org.apache.hadoop.mapred.TaskTracker
$TaskLauncher.run(TaskTracker.java:2176)


--
Drs. Job Tiel Groenestege
GridLine - Intranet en Zoeken

GridLine
Keizersgracht 520
1017 EK Amsterdam

www: http://www.gridline.nl
mail: job@gridline.nl
tel: +31 20 616 2050
fax: +31 20 616 2051

De inhoud van dit bericht en de eventueel daarbij behorende bijlagen zijn persoonlijk gericht aan en derhalve uitsluitend bestemd voor de geadresseerde. Zij kunnen gegevens met betrekking tot een derde bevatten. De ontvanger die niet de geadresseerde is, noch bevoegd is dit bericht namens geadresseerde te ontvangen, wordt verzocht de afzender onmiddellijk op de hoogte te stellen van de ontvangst. Elk gebruik van de inhoud van dit bericht en/of van de daarbij behorende bijlagen door een ander dan de geadresseerde is onrechtmatig jegens afzender respectievelijk de hiervoor bedoelde derde.

Search Discussions

  • Todd Lipcon at Feb 24, 2011 at 6:18 pm
    Hi Job,

    This seems CDH-specific, so I've moved the thread over to the cdh-users
    mailing list (BCC common-user)

    Thanks
    -Todd
    On Thu, Feb 24, 2011 at 2:52 AM, Job wrote:

    Hi all,

    This issue could very well be related to the Cloudera distribution
    (CDH3b4) I use, but maybe someone knows the solution:

    I configured a Job, something like this:

    Configuration conf = getConf();
    // ... set configuration
    conf.set("mapred.jar", localJarFile.toString())
    // tracker, zookeeper, hbase etc.


    Job job = new Job(conf);
    // map:
    job.setMapperClass(DataImportMap.class);
    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(Put.class);
    // reduce:

    TableMapReduceUtil.initTableReducerJob("MyTable",
    DataImportReduce.class, job);
    FileInputFormat.addInputPath(job, new Path(inputData));

    // execute:
    job.waitForCompletion(true);

    Now the server throws a strange exception below, see the stacktrace
    below.

    When i take look at the hdfs file system - through hdfs fuse - the file
    is there, it really is the jar that contains my mapred classes.

    Any clue wat goes wrong here?

    Thanks,
    Job


    ---------------------------------------------
    java.io.FileNotFoundException:
    File
    /var/lib/hadoop-0.20/cache/mapred/mapred/staging/job/.staging/job_201102241026_0002/job.jar
    does not exist.
    at

    org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
    at

    org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:157)
    at

    org.apache.hadoop.fs.LocalFileSystem.copyToLocalFile(LocalFileSystem.java:61)
    at
    org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1303)
    at

    org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
    at

    org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:381)
    at

    org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:371)
    at

    org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:198)
    at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1154)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at

    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
    at
    org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1129)
    at
    org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1055)
    at
    org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2212)
    at org.apache.hadoop.mapred.TaskTracker
    $TaskLauncher.run(TaskTracker.java:2176)


    --
    Drs. Job Tiel Groenestege
    GridLine - Intranet en Zoeken

    GridLine
    Keizersgracht 520
    1017 EK Amsterdam

    www: http://www.gridline.nl
    mail: job@gridline.nl
    tel: +31 20 616 2050
    fax: +31 20 616 2051

    De inhoud van dit bericht en de eventueel daarbij behorende bijlagen zijn
    persoonlijk gericht aan en derhalve uitsluitend bestemd voor de
    geadresseerde. Zij kunnen gegevens met betrekking tot een derde bevatten. De
    ontvanger die niet de geadresseerde is, noch bevoegd is dit bericht namens
    geadresseerde te ontvangen, wordt verzocht de afzender onmiddellijk op de
    hoogte te stellen van de ontvangst. Elk gebruik van de inhoud van dit
    bericht en/of van de daarbij behorende bijlagen door een ander dan de
    geadresseerde is onrechtmatig jegens afzender respectievelijk de hiervoor
    bedoelde derde.

    --
    Todd Lipcon
    Software Engineer, Cloudera

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedFeb 24, '11 at 10:52a
activeFeb 24, '11 at 6:18p
posts2
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Job: 1 post Todd Lipcon: 1 post

People

Translate

site design / logo © 2022 Grokbase