FAQ
Harsh! Thanks for the super-fast replies! I really appreciate it.


Actually, I had forgotten to redeploy the client configuration from
Cloudera Manager.

I just did that and now the YARN jobs starts, but it doesn't seem to be
progressing at all. The Map is at 0% and the Reduce is at 0%.

What could be happening now?

Why's it getting stuck?


[root@cdh4-cm ~]# hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep yarn_input
yarn_output 'dfs[a-z.]+'
12/08/27 19:16:49 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
12/08/27 19:16:49 INFO input.FileInputFormat: Total input paths to process
: 3
12/08/27 19:16:49 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
12/08/27 19:16:49 WARN snappy.LoadSnappy: Snappy native library is available
12/08/27 19:16:49 INFO snappy.LoadSnappy: Snappy native library loaded
12/08/27 19:16:49 INFO mapreduce.JobSubmitter: number of splits:3
12/08/27 19:16:49 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
12/08/27 19:16:49 WARN conf.Configuration: mapreduce.combine.class is
deprecated. Instead, use mapreduce.job.combine.class
12/08/27 19:16:49 WARN conf.Configuration: mapreduce.map.class is
deprecated. Instead, use mapreduce.job.map.class
12/08/27 19:16:49 WARN conf.Configuration: mapred.job.name is deprecated.
Instead, use mapreduce.job.name
12/08/27 19:16:49 WARN conf.Configuration: mapreduce.reduce.class is
deprecated. Instead, use mapreduce.job.reduce.class
12/08/27 19:16:49 WARN conf.Configuration: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
12/08/27 19:16:49 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
12/08/27 19:16:49 WARN conf.Configuration: mapreduce.outputformat.class is
deprecated. Instead, use mapreduce.job.outputformat.class
12/08/27 19:16:49 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
12/08/27 19:16:49 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
12/08/27 19:16:49 WARN conf.Configuration: mapred.working.dir is
deprecated. Instead, use mapreduce.job.working.dir
12/08/27 19:16:50 INFO mapred.YARNRunner: Job jar is not present. Not
adding any jar to the list of resources.
12/08/27 19:16:50 INFO mapred.ResourceMgrDelegate: Submitted application
application_1346092367798_0001 to ResourceManager at
cdh4-cm-vm0/108.166.81.199:8032
12/08/27 19:16:50 INFO mapreduce.Job: The url to track the job:
http://cdh4-cm:8088/proxy/application_1346092367798_0001/
12/08/27 19:16:50 INFO mapreduce.Job: Running job: job_1346092367798_0001
12/08/27 19:16:57 INFO mapreduce.Job: Job job_1346092367798_0001 running in
uber mode : false
12/08/27 19:16:57 INFO mapreduce.Job: *map 0% reduce 0%*


The web UI shows that all maps are stuck in a scheduled state with the task
unassigned right now. Why would they not be getting assigned properly?

The job has been running for 8 minutes now with no progress. Hmm...

- J

On Monday, August 27, 2012 3:24:24 PM UTC-4, Harsh J wrote:

Hi Jon,

YARN clients require a yarn-site.xml configured with address of the
RM, and a mapred-site.xml that carries at least this:

<property><name>mapreduce.framework.name</name><value>yarn</value></property>

On Tue, Aug 28, 2012 at 12:33 AM, Jon Ramos wrote:
Hello Cloudera Manager users,

I'm testing out YARN for the first time for my organization and am having
trouble running a simple YARN grep app. I was hoping somebody should help me
as I seem to be stuck. I used the example from the end of the Cloudera CDH4
Quick Start guide for this.


I have already loaded some XML files into HDFS here:

[root@cdh4-cm ~]# hadoop fs -ls yarn_input
Found 3 items
-rw-r--r-- 1 root supergroup 822 2012-08-27 18:39
yarn_input/core-site.xml
-rw-r--r-- 1 root supergroup 697 2012-08-27 18:39
yarn_input/hdfs-site.xml
-rw-r--r-- 1 root supergroup 1934 2012-08-27 18:39
yarn_input/mapred-site.xml


And I have properly set my environment variable:

[root@cdh4-cm ~]# echo $HADOOP_MAPRED_HOME
/usr/lib/hadoop-mapreduce


Here is the error I get when I try to run the example code:

Note: The Cloudera Manager UI shows that HDFS and YARN services are running
and healthy at the moment. I have also stopped the mapreduce1 service so
there is no incompatibility. Actually all other services hae been stopped.
[root@cdh4-cm ~]# hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep yarn_input
yarn_output 'dfs[a-z.]+'
12/08/27 18:53:41 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid
"mapreduce.jobtracker.address" configuration value for LocalJobRunner :
"cdh4-cm:8021"
12/08/27 18:53:41 ERROR security.UserGroupInformation:
PriviledgedActionException as:root (auth:SIMPLE)
cause:java.io.IOException:
Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1196)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1192)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1191)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1220)
at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
at org.apache.hadoop.examples.Grep.run(Grep.java:77)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)


What is going on here? How do I fix the "Invalid
"mapreduce.jobtracker.address" configuration value for LocalJobRunner :
"cdh4-cm:8021"???

Also, why is a JobTracker being used when I'm running a YARN example?
Shouldn't it be looking for a NodeManager or ResourceManager instead?

- J


--
Harsh J

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 9 | next ›
Discussion Overview
groupscm-users @
categorieshadoop
postedAug 27, '12 at 7:03p
activeAug 28, '12 at 6:57p
posts9
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase