FAQ
Hi folks. I just installed CDH 4 with Cloudera manager. The services I
installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
manager web portal, all of them are health. And I can also open 50030 job
tracker portal as well as 50070. Seems everything's fine.

But when I tried to execute a MR job as below. I cannot see anything in my
50030 portal (it's always empty). And from the verbose below, it seems that
it's a local job instead of MR. I tried PIG with MR mode, the same problem.
Can anyone tell me how can I run my job on cluster or fix portal issue if
it is? Thx in advance!


$ <hadoop@namenode:/usr/lib/hadoop-mapreduce$> sudo -u hdfs hadoop jar
hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar wordcount
hdfs://namenode:8020/test/test \test\result
13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id
13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process
: 1
13/02/09 11:48:22 INFO mapred.JobClient: Running job: *job_local_0001*
13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
null
13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
13/02/09 11:48:23 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
13/02/09 11:48:23 INFO mapred.LocalJobRunner:
13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
done. And is in the process of commiting
13/02/09 11:48:23 INFO mapred.LocalJobRunner:
13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
done.
13/02/09 11:48:23 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
13/02/09 11:48:23 INFO mapred.LocalJobRunner:
13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 26 bytes
13/02/09 11:48:23 INFO mapred.LocalJobRunner:
13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
done. And is in the process of commiting
13/02/09 11:48:23 INFO mapred.LocalJobRunner:
13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
allowed to commit now
13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_000000_0' to testresult
13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
done.
13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
read=285550
13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
written=458244
13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
operations=0
13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
operations=0
13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
operations=0
13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
written=16
13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
operations=9
13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
operations=0
13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
operations=3
13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
snapshot=0
13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=0
13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
(bytes)=324542464

--

Search Discussions

  • Harsh J at Feb 9, 2013 at 5:36 pm
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at
    https://ccp.cloudera.com/display/ENT41DOC/Deploying+Client+Configuration+Files?
    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J

    --
  • Balto Jenna at Feb 10, 2013 at 7:37 am
    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at

    https://ccp.cloudera.com/display/ENT41DOC/Deploying+Client+Configuration+Files?

    Doing that will solve your issue.

    On Sat, Feb 9, 2013 at 10:26 PM, <balto...@gmail.com <javascript:>>
    wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --
  • Deepak Gattala at Nov 6, 2013 at 1:31 am
    Hi Hadoop Experts,

    I am in need in help,
    Got into the same situation
    when i run the sqoop jobs its not visible in the Job tracker UI and it says
    its running locally like below, all otehr count(*) jobs in Hive and others
    just work fine but only the sqooop MR jobs importing and exporting dont
    work on cluster and works only locally.

    13/11/05 19:30:24 INFO mapred.LocalJobRunner:
    13/11/05 19:30:25 DEBUG mapreduce.AutoProgressMapper: Auto-progress thread
    reporting progress
    13/11/05 19:30:27 INFO mapred.LocalJobRunner:
    13/11/05 19:30:30 INFO mapred.LocalJobRunner:


    Please advise.

    Thanks
    Deepak Gattala
    On Sunday, February 10, 2013 1:37:49 AM UTC-6, balto...@gmail.com wrote:

    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at

    https://ccp.cloudera.com/display/ENT41DOC/Deploying+Client+Configuration+Files?

    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task
    'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task
    'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Anurag Tangri at Nov 6, 2013 at 1:56 am
    Looks like you are missing mapred-site.xml on your client/gateway.

    Check your Hadoop conf dir.

    Thanks,
    Anurag Tangri
    On Nov 5, 2013, at 5:31 PM, Deepak Gattala wrote:

    Hi Hadoop Experts,

    I am in need in help,
    Got into the same situation
    when i run the sqoop jobs its not visible in the Job tracker UI and it says its running locally like below, all otehr count(*) jobs in Hive and others just work fine but only the sqooop MR jobs importing and exporting dont work on cluster and works only locally.

    13/11/05 19:30:24 INFO mapred.LocalJobRunner:
    13/11/05 19:30:25 DEBUG mapreduce.AutoProgressMapper: Auto-progress thread reporting progress
    13/11/05 19:30:27 INFO mapred.LocalJobRunner:
    13/11/05 19:30:30 INFO mapred.LocalJobRunner:


    Please advise.

    Thanks
    Deepak Gattala
    On Sunday, February 10, 2013 1:37:49 AM UTC-6, balto...@gmail.com wrote:
    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at
    https://ccp.cloudera.com/display/ENT41DOC/Deploying+Client+Configuration+Files?
    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Deepak Gattala at Nov 6, 2013 at 2:29 am
    I am using cloudera manager how can I miss it. I never did any manual
    changes.

    Where I have to see for this file where you mean its missing in what
    directory.

    I tried to compare it cloudera vm and everything looks correct only.

    Where you think the entry to the directory is missing can you be more clear
    please I am not that expert in this area.

    Thanks a million
    Deepak Gattala
    On Nov 5, 2013 7:57 PM, "Anurag Tangri" wrote:

    Looks like you are missing mapred-site.xml on your client/gateway.

    Check your Hadoop conf dir.

    Thanks,
    Anurag Tangri

    On Nov 5, 2013, at 5:31 PM, Deepak Gattala wrote:

    Hi Hadoop Experts,

    I am in need in help,
    Got into the same situation
    when i run the sqoop jobs its not visible in the Job tracker UI and it
    says its running locally like below, all otehr count(*) jobs in Hive and
    others just work fine but only the sqooop MR jobs importing and exporting
    dont work on cluster and works only locally.

    13/11/05 19:30:24 INFO mapred.LocalJobRunner:
    13/11/05 19:30:25 DEBUG mapreduce.AutoProgressMapper: Auto-progress thread
    reporting progress
    13/11/05 19:30:27 INFO mapred.LocalJobRunner:
    13/11/05 19:30:30 INFO mapred.LocalJobRunner:


    Please advise.

    Thanks
    Deepak Gattala
    On Sunday, February 10, 2013 1:37:49 AM UTC-6, balto...@gmail.com wrote:

    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at
    https://ccp.cloudera.com/display/ENT41DOC/Deploying+
    Client+Configuration+Files?
    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --
    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.

    --

    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Deepak Gattala at Nov 6, 2013 at 2:43 am
    to be more clear,

    I will kick off this sqoop job from the edge node and edge node i snot the
    job tacker and neither part of the data node.

    so the hdfs site and mapred or any thing will be on it except the sqoop
    site and some like hue ini and etc.

    is that because of that if yes how can i resolve it, if not please advise
    the correct approach really appreciate all your help.

    Thanks
    Deepak Gattala


    On Tue, Nov 5, 2013 at 8:29 PM, Deepak Gattala wrote:

    I am using cloudera manager how can I miss it. I never did any manual
    changes.

    Where I have to see for this file where you mean its missing in what
    directory.

    I tried to compare it cloudera vm and everything looks correct only.

    Where you think the entry to the directory is missing can you be more
    clear please I am not that expert in this area.

    Thanks a million
    Deepak Gattala
    On Nov 5, 2013 7:57 PM, "Anurag Tangri" wrote:

    Looks like you are missing mapred-site.xml on your client/gateway.

    Check your Hadoop conf dir.

    Thanks,
    Anurag Tangri

    On Nov 5, 2013, at 5:31 PM, Deepak Gattala wrote:

    Hi Hadoop Experts,

    I am in need in help,
    Got into the same situation
    when i run the sqoop jobs its not visible in the Job tracker UI and it
    says its running locally like below, all otehr count(*) jobs in Hive and
    others just work fine but only the sqooop MR jobs importing and exporting
    dont work on cluster and works only locally.

    13/11/05 19:30:24 INFO mapred.LocalJobRunner:
    13/11/05 19:30:25 DEBUG mapreduce.AutoProgressMapper: Auto-progress
    thread reporting progress
    13/11/05 19:30:27 INFO mapred.LocalJobRunner:
    13/11/05 19:30:30 INFO mapred.LocalJobRunner:


    Please advise.

    Thanks
    Deepak Gattala
    On Sunday, February 10, 2013 1:37:49 AM UTC-6, balto...@gmail.com wrote:

    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at
    https://ccp.cloudera.com/display/ENT41DOC/Deploying+
    Client+Configuration+Files?
    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it
    seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal
    issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer =
    79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task
    attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --
    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.

    --

    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.
  • Deepak Gattala at Nov 6, 2013 at 3:07 am
    thanks alot Anurag,

    i got ti resolved you been a great help really appreciate all the direction
    you have provided.

    Thanks
    Deepak Gattala

    On Tue, Nov 5, 2013 at 7:56 PM, Anurag Tangri wrote:

    Looks like you are missing mapred-site.xml on your client/gateway.

    Check your Hadoop conf dir.

    Thanks,
    Anurag Tangri

    On Nov 5, 2013, at 5:31 PM, Deepak Gattala wrote:

    Hi Hadoop Experts,

    I am in need in help,
    Got into the same situation
    when i run the sqoop jobs its not visible in the Job tracker UI and it
    says its running locally like below, all otehr count(*) jobs in Hive and
    others just work fine but only the sqooop MR jobs importing and exporting
    dont work on cluster and works only locally.

    13/11/05 19:30:24 INFO mapred.LocalJobRunner:
    13/11/05 19:30:25 DEBUG mapreduce.AutoProgressMapper: Auto-progress thread
    reporting progress
    13/11/05 19:30:27 INFO mapred.LocalJobRunner:
    13/11/05 19:30:30 INFO mapred.LocalJobRunner:


    Please advise.

    Thanks
    Deepak Gattala
    On Sunday, February 10, 2013 1:37:49 AM UTC-6, balto...@gmail.com wrote:

    Resolved as your way. Thanks very much.

    在 2013年2月9日星期六UTC-8上午9时36分03秒,Harsh J写道:
    Hi,

    Have you deployed cluster-client configurations to your machine (where
    you invoke "hadoop jar") as detailed at
    https://ccp.cloudera.com/display/ENT41DOC/Deploying+
    Client+Configuration+Files?
    Doing that will solve your issue.
    On Sat, Feb 9, 2013 at 10:26 PM, wrote:
    Hi folks. I just installed CDH 4 with Cloudera manager. The services I
    installed are hdfs, hbase, mapreduce, oozie, zookeeper. From Cloudera
    manager web portal, all of them are health. And I can also open 50030 job
    tracker portal as well as 50070. Seems everything's fine.

    But when I tried to execute a MR job as below. I cannot see anything in my
    50030 portal (it's always empty). And from the verbose below, it seems that
    it's a local job instead of MR. I tried PIG with MR mode, the same problem.
    Can anyone tell me how can I run my job on cluster or fix portal issue if it
    is? Thx in advance!


    $ sudo -u hdfs hadoop jar hadoop-mapreduce-examples-2.0.0-cdh4.1.2.jar
    wordcount hdfs://namenode:8020/test/test \test\result
    13/02/09 11:48:22 WARN conf.Configuration: session.id is deprecated.
    Instead, use dfs.metrics.session-id
    13/02/09 11:48:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with
    processName=JobTracker, sessionId=
    13/02/09 11:48:22 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    13/02/09 11:48:22 INFO input.FileInputFormat: Total input paths to process :
    1
    13/02/09 11:48:22 INFO mapred.JobClient: Running job: job_local_0001
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter set in config
    null
    13/02/09 11:48:22 INFO mapred.LocalJobRunner: OutputCommitter is
    org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO util.ProcessTree: setsid exited with exit code 0
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@ec0a9f9
    13/02/09 11:48:23 INFO mapred.MapTask: io.sort.mb = 100
    13/02/09 11:48:23 INFO mapred.MapTask: data buffer = 79691776/99614720
    13/02/09 11:48:23 INFO mapred.MapTask: record buffer = 262144/327680
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.MapTask: Starting flush of map output
    13/02/09 11:48:23 INFO mapred.MapTask: Finished spill 0
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
    done.
    13/02/09 11:48:23 WARN mapreduce.Counters: Group
    org.apache.hadoop.mapred.Task$Counter is deprecated. Use
    org.apache.hadoop.mapreduce.TaskCounter instead
    13/02/09 11:48:23 INFO mapred.Task: Using ResourceCalculatorPlugin :
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin@25cbfa42
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Merger: Merging 1 sorted segments
    13/02/09 11:48:23 INFO mapred.Merger: Down to the last merge-pass, with 1
    segments left of total size: 26 bytes
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is
    done. And is in the process of commiting
    13/02/09 11:48:23 INFO mapred.LocalJobRunner:
    13/02/09 11:48:23 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is
    allowed to commit now
    13/02/09 11:48:23 INFO output.FileOutputCommitter: Saved output of task
    'attempt_local_0001_r_000000_0' to testresult
    13/02/09 11:48:23 INFO mapred.LocalJobRunner: reduce > reduce
    13/02/09 11:48:23 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0'
    done.
    13/02/09 11:48:23 INFO mapred.JobClient: map 100% reduce 100%
    13/02/09 11:48:23 INFO mapred.JobClient: Job complete: job_local_0001
    13/02/09 11:48:23 INFO mapred.JobClient: Counters: 25
    13/02/09 11:48:23 INFO mapred.JobClient: File System Counters
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    read=285550
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of bytes
    written=458244
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: FILE: Number of write
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes read=24
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of bytes
    written=16
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of read
    operations=9
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    13/02/09 11:48:23 INFO mapred.JobClient: HDFS: Number of write
    operations=3
    13/02/09 11:48:23 INFO mapred.JobClient: Map-Reduce Framework
    13/02/09 11:48:23 INFO mapred.JobClient: Map input records=1
    13/02/09 11:48:23 INFO mapred.JobClient: Map output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Map output bytes=20
    13/02/09 11:48:23 INFO mapred.JobClient: Input split bytes=95
    13/02/09 11:48:23 INFO mapred.JobClient: Combine input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Combine output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input groups=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce shuffle bytes=0
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce input records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Reduce output records=2
    13/02/09 11:48:23 INFO mapred.JobClient: Spilled Records=4
    13/02/09 11:48:23 INFO mapred.JobClient: CPU time spent (ms)=0
    13/02/09 11:48:23 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=0
    13/02/09 11:48:23 INFO mapred.JobClient: Total committed heap usage
    (bytes)=324542464

    --



    --
    Harsh J
    --
    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.

    --

    ---
    You received this message because you are subscribed to the Google Groups
    "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit
    https://groups.google.com/a/cloudera.org/groups/opt_out.
    --

    ---
    You received this message because you are subscribed to the Google Groups "CDH Users" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
    For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedFeb 9, '13 at 4:56p
activeNov 6, '13 at 3:07a
posts8
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase