FAQ
Hi,

I am a beginner user of a hadoop.

almost examples to learn hadoop suggest to use a jar style to use a
hadoop framework.
(like workcount.jar)

in this case, I could find out a job history.

but, if I execute my application as a java application (not a jar file).

I could't find out job histories in a jobtracker page.

and also I've set up two nodes as a hadoop cluster.

however, my java application looks like using just a single node, not a
two nodes to run my own sample.

so.....

to track my job history, do I need to create jar files always?

--
Junyoung Kim (juneng603@gmail.com)

Search Discussions

  • Aman at Jan 24, 2011 at 5:30 pm
    Not 100% sure of what your java program does but it looks like in your java
    application, you are not using Job Tracker in any way. It will help of you
    can post the nature of your java program


    Jun Young Kim wrote:
    Hi,

    I am a beginner user of a hadoop.

    almost examples to learn hadoop suggest to use a jar style to use a
    hadoop framework.
    (like workcount.jar)

    in this case, I could find out a job history.

    but, if I execute my application as a java application (not a jar file).

    I could't find out job histories in a jobtracker page.

    and also I've set up two nodes as a hadoop cluster.

    however, my java application looks like using just a single node, not a
    two nodes to run my own sample.

    so.....

    to track my job history, do I need to create jar files always?

    --
    Junyoung Kim (juneng603@gmail.com)

    --
    View this message in context: http://lucene.472066.n3.nabble.com/I-couldn-t-find-out-job-histories-in-a-jobtracker-page-tp2318366p2321229.html
    Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
  • Jun Young Kim at Jan 25, 2011 at 1:11 am
    my application is quite simple.
    a) it reads from files in a directory.
    b) it calls a map & reduce function to compare it's data of input
    files.
    c) it writes a result of comparison of input files to a output file.

    here is my code.
    .. job class..
    Configuration sConf = new Configuration();
    GenericOptionsParser sParser = new GenericOptionsParser(sConf,
    aArgs);
    Job sJob = null;

    String[] sOtherArgs = sParser.getRemainingArgs();

    sJob = new Job(sConf, "EPComparatorJob");

    log.info("sJob = " + sJob);
    sJob.setJarByClass(EPComparatorJob.class);

    sJob.setMapOutputKeyClass(Text.class);
    sJob.setMapOutputValueClass(Text.class);

    sJob.setOutputKeyClass(Text.class);
    sJob.setOutputValueClass(Text.class);
    sJob.setInputFormatClass(TextInputFormat.class);

    if (sOtherArgs.length != 2) {
    printUsage();
    System.exit(1);
    }

    log.info("setInput & Output paths");
    FileInputFormat.setInputPaths(sJob, new Path(sOtherArgs[0]));
    FileOutputFormat.setOutputPath(sJob, new Path(sOtherArgs[1]));

    MultipleOutputs.addNamedOutput(sJob,
    HadoopConfig.INSERT_OUTPUT_NAME, TextOutputFormat.class, Text.class,
    Text.class);
    MultipleOutputs.addNamedOutput(sJob,
    HadoopConfig.DELETE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
    Text.class);
    MultipleOutputs.addNamedOutput(sJob,
    HadoopConfig.UPDATE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
    Text.class);
    MultipleOutputs.addNamedOutput(sJob,
    HadoopConfig.NOTCHANGE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
    Text.class);

    log.info("setMapperClass");
    sJob.setMapperClass(EPComparatorMapper.class);

    log.info("setReducerClass");
    sJob.setReducerClass(EPComparatorReducer.class);

    log.info("setNumReduceTasks");
    sJob.setNumReduceTasks(REDUCE_MAPTASKS_COUNTS);

    return (sJob.waitForCompletion(true) == true ? 0 : 1);

    .. map class..
    ...
    protected void map(WritableComparable<Text> aKey, Text aValue,
    Context aContext) throws IOException, InterruptedException {
    String info = aValue.toString();
    String[] fields = info.split(HadoopConfig.EP_DATA_DELIMETER, 2);

    // input file name
    Path file = ((FileSplit)aContext.getInputSplit()).getPath();

    String key = fields[0].trim();
    String value = fields[1].trim() +
    HadoopConfig.EP_DATA_DELIMETER + file;

    aContext.write(new Text(key), new Text(value));
    };
    ...

    .. reduce class..
    ...
    protected void reduce(WritableComparable<Text> key, Iterable<Text>
    values, Context context) throws IOException, InterruptedException {
    String[] ret = getComparedResult(key, values, context);
    String code = ret[0];
    String keyMapid = ret[1];
    String valueInfo = ret[2];

    multipleOutputs.write(new Text(code), new Text(keyMapid +
    HadoopConfig.EP_DATA_DELIMETER + valueInfo), getOutputFileName(code));
    }
    ...


    I got the email from another user of a hadoop about this problem.
    the point of the email is we need to deploy an application as a jar
    style to use job trackers, not a java application itself.
    because to run map&reduce functions on slaves(cluster), we NEED to run a
    hadoop with a jar.

    thanks.

    Junyoung Kim (juneng603@gmail.com)

    On 01/25/2011 02:30 AM, Aman wrote:
    Not 100% sure of what your java program does but it looks like in your java
    application, you are not using Job Tracker in any way. It will help of you
    can post the nature of your java program


    Jun Young Kim wrote:
    Hi,

    I am a beginner user of a hadoop.

    almost examples to learn hadoop suggest to use a jar style to use a
    hadoop framework.
    (like workcount.jar)

    in this case, I could find out a job history.

    but, if I execute my application as a java application (not a jar file).

    I could't find out job histories in a jobtracker page.

    and also I've set up two nodes as a hadoop cluster.

    however, my java application looks like using just a single node, not a
    two nodes to run my own sample.

    so.....

    to track my job history, do I need to create jar files always?

    --
    Junyoung Kim (juneng603@gmail.com)

  • Harsh J at Jan 25, 2011 at 3:08 am
    A job will appear on the JobTracker if it has been submitted to it. I
    believe that your program is probably running in a local mode (as set
    by defaults, not by your configuration mostly) and not actually
    submitting to the live JobTracker. Could you verify, by the logs
    thrown out upon execution of your program, that it is submitting to
    the right JobTracker (A live daemon, and not "local" job runners)?

    Using a jar or not for submissions, does not affect the visibility of
    job details on the JT's web-ui. All that matters is that the
    configuration loaded at construction is your true config and not
    defaults. You could also set "mapred.job.tracker" and
    "fs.default.name" manually, for instance.
    On Mon, Jan 24, 2011 at 3:00 PM, Jun Young Kim wrote:
    Hi,

    I am a beginner user of a hadoop.

    almost examples to learn hadoop suggest to use a jar style to use  a hadoop
    framework.
    (like workcount.jar)

    in this case, I could find out a job history.

    but, if I execute my application as a java application (not a jar file).

    I could't find out job histories in a jobtracker page.

    and also I've set up two nodes as a hadoop cluster.

    however, my java application looks like using just a single node, not a two
    nodes to run my own sample.

    so.....

    to track my job history, do I need to create jar files always?

    --
    Junyoung Kim (juneng603@gmail.com)


    --
    Harsh J
    www.harshj.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 24, '11 at 9:30a
activeJan 25, '11 at 3:08a
posts4
users3
websitehadoop.apache.org...
irc#hadoop

3 users in discussion

Jun Young Kim: 2 posts Aman: 1 post Harsh J: 1 post

People

Translate

site design / logo © 2022 Grokbase