Hai guys !

I have set up 5 node cluster with each of them in different racks.

I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder
using
Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
I ran some jobs on cluster and used copy of /usr/local/hadoop/logs/history
folder of *master *node as input to TraceBuilder.
Arguments: /home/arun/job.json /home/arun/topology.json
/home/ubuntu/Documents/testlog

But i get
11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while
processing the [possibly virtual] file
master_1324011575958_job_201112161029_0001_hduser_word+count within Path
file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count

java.lang.NullPointerException
at
org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)

at
org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
at
org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)

at
org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at
org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while
processing the [possibly virtual] file
master_1324011575958_job_201112161029_0004_hduser_word+count within Path
file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count

java.lang.NullPointerException
at
org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)

at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde
.........
....................

It generates job trace json file but the topology trace json file doesn't
have 5 node's info and is like this :

{
"name" : "<root>",
"children" : [ ]
}
Which history folder do i need to use ? masters or slaves ? how do i use it ?
I have a single node cluster from which i got logs and gave input
TraceBuilder and it works.
How do i run tracebuilder from terminal and do i need to use only from
version 21 ?

Can anyone help me out ?


Thanks,
Arun

Search Discussions

  • Ravi Gummadi at Dec 16, 2011 at 6:59 am
    Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to trunk yet. Feel free to try it out while it gets reviewed and committed.

    -Ravi
    ________________________________________
    From: arun k [arunk786@gmail.com]
    Sent: Friday, December 16, 2011 12:22 PM
    To: mapreduce-user@hadoop.apache.org
    Subject: Generating job and topology traces from history folder of multinode cluster using Rumen

    Hai guys !

    I have set up 5 node cluster with each of them in different racks.

    I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder using
    Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
    I ran some jobs on cluster and used copy of /usr/local/hadoop/logs/history folder of master node as input to TraceBuilder.
    Arguments: /home/arun/job.json /home/arun/topology.json /home/ubuntu/Documents/testlog

    But i get
    11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the [possibly virtual] file master_1324011575958_job_201112161029_0001_hduser_word+count within Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
    java.lang.NullPointerException
    at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
    at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
    at org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
    at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
    at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
    11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the [possibly virtual] file master_1324011575958_job_201112161029_0004_hduser_word+count within Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
    java.lang.NullPointerException
    at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
    at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde .........
    ....................

    It generates job trace json file but the topology trace json file doesn't have 5 node's info and is like this :

    {
    "name" : "<root>",
    "children" : [ ]
    }
    Which history folder do i need to use ? masters or slaves ? how do i use it ?
    I have a single node cluster from which i got logs and gave input TraceBuilder and it works.
    How do i run tracebuilder from terminal and do i need to use only from version 21 ?
    Can anyone help me out ?


    Thanks,
    Arun
  • Arun k at Dec 16, 2011 at 3:02 pm
    Ravi,

    Thanks for the info.


    Arun
    On Fri, Dec 16, 2011 at 12:27 PM, Ravi Gummadi wrote:

    Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to
    trunk yet. Feel free to try it out while it gets reviewed and committed.

    -Ravi
    ________________________________________
    From: arun k [arunk786@gmail.com]
    Sent: Friday, December 16, 2011 12:22 PM
    To: mapreduce-user@hadoop.apache.org
    Subject: Generating job and topology traces from history folder of
    multinode cluster using Rumen

    Hai guys !

    I have set up 5 node cluster with each of them in different racks.

    I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder
    using
    Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
    I ran some jobs on cluster and used copy of
    /usr/local/hadoop/logs/history folder of master node as input to
    TraceBuilder.
    Arguments: /home/arun/job.json /home/arun/topology.json
    /home/ubuntu/Documents/testlog

    But i get
    11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
    library for your platform... using builtin-java classes where applicable
    11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while
    processing the [possibly virtual] file
    master_1324011575958_job_201112161029_0001_hduser_word+count within Path
    file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
    java.lang.NullPointerException
    at
    org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
    at
    org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
    at
    org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
    at
    org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
    at
    org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
    11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while
    processing the [possibly virtual] file
    master_1324011575958_job_201112161029_0004_hduser_word+count within Path
    file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
    java.lang.NullPointerException
    at
    org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
    at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde
    .........
    ....................

    It generates job trace json file but the topology trace json file doesn't
    have 5 node's info and is like this :

    {
    "name" : "<root>",
    "children" : [ ]
    }
    Which history folder do i need to use ? masters or slaves ? how do i use it ?
    I have a single node cluster from which i got logs and gave input
    TraceBuilder and it works.
    How do i run tracebuilder from terminal and do i need to use only from
    version 21 ?

    Can anyone help me out ?


    Thanks,
    Arun

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmapreduce-user @
categorieshadoop
postedDec 16, '11 at 6:52a
activeDec 16, '11 at 3:02p
posts3
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Arun k: 2 posts Ravi Gummadi: 1 post

People

Translate

site design / logo © 2022 Grokbase