FAQ
Hi,

I have been using an Ubuntu 12.04 distribution as a CDH4 Hadoop testbed for
several months now without problems. I'd like to upgrade to Ubuntu 12.04.1
(a minor change) so I created a fresh virtual machine and followed the same
set of instructions for installing CDH4 in pseudo-distributed mode:

https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

I can verify that I used the same instructions because I have a very long
~/.history file and I can see that what I typed months ago matches the
webpage.

The Ubuntu 12.04 VM has Oracle JDK version 1.6.0_34:

% java -version
java version "1.6.0_34"
Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)

and the Ubuntu 12.04.1 VM has Oracle JDK version 1.6.0_37:

% java -version
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

Both are just slightly above the CDH4 recommended Java VM (1.6.0_31).
However, whereas the first version has been used in hundreds of successful
map-reduce jobs, the second version always throws NullPointerExceptions
when transitioning from the mappers to the reducers. I've included output
from a sample job below (from the "grep" test recommended in the
instructions; I've also written simple streaming jobs in Python and they
all fail the same way: NullPointerExceptions when the reducers start).

Does this make sense to anyone? Is there anything fragile in the
hadoop-0.20-conf-pseudo package?

Thanks,
-- Jim

% /usr/bin/hadoop jar /usr/lib/hadoop-0.20-
mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'
12/10/29 20:38:32 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/10/29 20:38:32 WARN snappy.LoadSnappy: Snappy native library is available
12/10/29 20:38:32 INFO snappy.LoadSnappy: Snappy native library loaded
12/10/29 20:38:32 INFO mapred.FileInputFormat: Total input paths to process
: 3
12/10/29 20:38:33 INFO mapred.JobClient: Running job: job_201210292036_0001
12/10/29 20:38:34 INFO mapred.JobClient: map 0% reduce 0%
12/10/29 20:38:41 INFO mapred.JobClient: map 66% reduce 0%
12/10/29 20:38:44 INFO mapred.JobClient: map 100% reduce 0%
12/10/29 20:38:45 INFO mapred.JobClient: Task Id :
attempt_201210292036_0001_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

12/10/29 20:38:49 INFO mapred.JobClient: Task Id :
attempt_201210292036_0001_r_000000_1, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

12/10/29 20:38:54 INFO mapred.JobClient: Task Id :
attempt_201210292036_0001_r_000000_2, Status : FAILED
Error: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

12/10/29 20:39:01 INFO mapred.JobClient: Job complete: job_201210292036_0001
12/10/29 20:39:01 INFO mapred.JobClient: Counters: 30
12/10/29 20:39:01 INFO mapred.JobClient: File System Counters
12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of bytes read=0
12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of bytes
written=537768
12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of read
operations=0
12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of large read
operations=0
12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of write
operations=0
12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of bytes read=4642
12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of bytes written=0
12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of read
operations=6
12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of large read
operations=0
12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of write
operations=0
12/10/29 20:39:01 INFO mapred.JobClient: Job Counters
12/10/29 20:39:01 INFO mapred.JobClient: Failed reduce tasks=1
12/10/29 20:39:01 INFO mapred.JobClient: Launched map tasks=3
12/10/29 20:39:01 INFO mapred.JobClient: Launched reduce tasks=4
12/10/29 20:39:01 INFO mapred.JobClient: Data-local map tasks=3
12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all maps
in occupied slots (ms)=13745
12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all
reduces in occupied slots (ms)=8395
12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
12/10/29 20:39:01 INFO mapred.JobClient: Map-Reduce Framework
12/10/29 20:39:01 INFO mapred.JobClient: Map input records=122
12/10/29 20:39:01 INFO mapred.JobClient: Map output records=6
12/10/29 20:39:01 INFO mapred.JobClient: Map output bytes=186
12/10/29 20:39:01 INFO mapred.JobClient: Input split bytes=326
12/10/29 20:39:01 INFO mapred.JobClient: Combine input records=6
12/10/29 20:39:01 INFO mapred.JobClient: Combine output records=6
12/10/29 20:39:01 INFO mapred.JobClient: Spilled Records=6
12/10/29 20:39:01 INFO mapred.JobClient: CPU time spent (ms)=1380
12/10/29 20:39:01 INFO mapred.JobClient: Physical memory (bytes)
snapshot=623812608
12/10/29 20:39:01 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=2937225216
12/10/29 20:39:01 INFO mapred.JobClient: Total committed heap usage
(bytes)=497614848
12/10/29 20:39:01 INFO mapred.JobClient:
org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
12/10/29 20:39:01 INFO mapred.JobClient: BYTES_READ=4316
12/10/29 20:39:01 INFO mapred.JobClient: Job Failed: NA
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

--

Search Discussions

  • Harsh J at Oct 30, 2012 at 5:33 am
    Hey Jim,

    Do any of your hostnames in the cluster carry underscores in them
    and/or not-conform to RFC 1123?

    On Tue, Oct 30, 2012 at 8:35 AM, Jim Pivarski
    wrote:
    Hi,

    I have been using an Ubuntu 12.04 distribution as a CDH4 Hadoop testbed for
    several months now without problems. I'd like to upgrade to Ubuntu 12.04.1
    (a minor change) so I created a fresh virtual machine and followed the same
    set of instructions for installing CDH4 in pseudo-distributed mode:

    https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

    I can verify that I used the same instructions because I have a very long
    ~/.history file and I can see that what I typed months ago matches the
    webpage.

    The Ubuntu 12.04 VM has Oracle JDK version 1.6.0_34:

    % java -version
    java version "1.6.0_34"
    Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
    Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)

    and the Ubuntu 12.04.1 VM has Oracle JDK version 1.6.0_37:

    % java -version
    java version "1.6.0_37"
    Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
    Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)

    Both are just slightly above the CDH4 recommended Java VM (1.6.0_31).
    However, whereas the first version has been used in hundreds of successful
    map-reduce jobs, the second version always throws NullPointerExceptions when
    transitioning from the mappers to the reducers. I've included output from a
    sample job below (from the "grep" test recommended in the instructions; I've
    also written simple streaming jobs in Python and they all fail the same way:
    NullPointerExceptions when the reducers start).

    Does this make sense to anyone? Is there anything fragile in the
    hadoop-0.20-conf-pseudo package?

    Thanks,
    -- Jim

    % /usr/bin/hadoop jar /usr/lib/hadoop-0.20-
    mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'
    12/10/29 20:38:32 WARN mapred.JobClient: Use GenericOptionsParser for
    parsing the arguments. Applications should implement Tool for the same.
    12/10/29 20:38:32 WARN snappy.LoadSnappy: Snappy native library is available
    12/10/29 20:38:32 INFO snappy.LoadSnappy: Snappy native library loaded
    12/10/29 20:38:32 INFO mapred.FileInputFormat: Total input paths to process
    : 3
    12/10/29 20:38:33 INFO mapred.JobClient: Running job: job_201210292036_0001
    12/10/29 20:38:34 INFO mapred.JobClient: map 0% reduce 0%
    12/10/29 20:38:41 INFO mapred.JobClient: map 66% reduce 0%
    12/10/29 20:38:44 INFO mapred.JobClient: map 100% reduce 0%
    12/10/29 20:38:45 INFO mapred.JobClient: Task Id :
    attempt_201210292036_0001_r_000000_0, Status : FAILED
    Error: java.lang.NullPointerException
    at
    java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

    12/10/29 20:38:49 INFO mapred.JobClient: Task Id :
    attempt_201210292036_0001_r_000000_1, Status : FAILED
    Error: java.lang.NullPointerException
    at
    java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

    12/10/29 20:38:54 INFO mapred.JobClient: Task Id :
    attempt_201210292036_0001_r_000000_2, Status : FAILED
    Error: java.lang.NullPointerException
    at
    java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2883)
    at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2794)

    12/10/29 20:39:01 INFO mapred.JobClient: Job complete: job_201210292036_0001
    12/10/29 20:39:01 INFO mapred.JobClient: Counters: 30
    12/10/29 20:39:01 INFO mapred.JobClient: File System Counters
    12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of bytes read=0
    12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of bytes
    written=537768
    12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of read
    operations=0
    12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of large read
    operations=0
    12/10/29 20:39:01 INFO mapred.JobClient: FILE: Number of write
    operations=0
    12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of bytes read=4642
    12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of bytes written=0
    12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of read
    operations=6
    12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of large read
    operations=0
    12/10/29 20:39:01 INFO mapred.JobClient: HDFS: Number of write
    operations=0
    12/10/29 20:39:01 INFO mapred.JobClient: Job Counters
    12/10/29 20:39:01 INFO mapred.JobClient: Failed reduce tasks=1
    12/10/29 20:39:01 INFO mapred.JobClient: Launched map tasks=3
    12/10/29 20:39:01 INFO mapred.JobClient: Launched reduce tasks=4
    12/10/29 20:39:01 INFO mapred.JobClient: Data-local map tasks=3
    12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all maps in
    occupied slots (ms)=13745
    12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all reduces
    in occupied slots (ms)=8395
    12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all maps
    waiting after reserving slots (ms)=0
    12/10/29 20:39:01 INFO mapred.JobClient: Total time spent by all reduces
    waiting after reserving slots (ms)=0
    12/10/29 20:39:01 INFO mapred.JobClient: Map-Reduce Framework
    12/10/29 20:39:01 INFO mapred.JobClient: Map input records=122
    12/10/29 20:39:01 INFO mapred.JobClient: Map output records=6
    12/10/29 20:39:01 INFO mapred.JobClient: Map output bytes=186
    12/10/29 20:39:01 INFO mapred.JobClient: Input split bytes=326
    12/10/29 20:39:01 INFO mapred.JobClient: Combine input records=6
    12/10/29 20:39:01 INFO mapred.JobClient: Combine output records=6
    12/10/29 20:39:01 INFO mapred.JobClient: Spilled Records=6
    12/10/29 20:39:01 INFO mapred.JobClient: CPU time spent (ms)=1380
    12/10/29 20:39:01 INFO mapred.JobClient: Physical memory (bytes)
    snapshot=623812608
    12/10/29 20:39:01 INFO mapred.JobClient: Virtual memory (bytes)
    snapshot=2937225216
    12/10/29 20:39:01 INFO mapred.JobClient: Total committed heap usage
    (bytes)=497614848
    12/10/29 20:39:01 INFO mapred.JobClient:
    org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
    12/10/29 20:39:01 INFO mapred.JobClient: BYTES_READ=4316
    12/10/29 20:39:01 INFO mapred.JobClient: Job Failed: NA
    java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
    at org.apache.hadoop.examples.Grep.run(Grep.java:69)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.examples.Grep.main(Grep.java:93)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at
    org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
    at
    org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

    --



    --
    Harsh J

    --
  • Jim Pivarski at Oct 30, 2012 at 7:29 pm

    On Tue, Oct 30, 2012 at 12:32 AM, Harsh J wrote:

    Do any of your hostnames in the cluster carry underscores in them
    and/or not-conform to RFC 1123?
    Yes, that seems to have been it, thank you!

    When I installed the distribution on the new VM, I entered a hostname that
    included periods (.) to indicate version number. The resulting VM was
    truncated at the position of the first period (that is, I wanted
    "ubuntu-12.04.1" but I got "ubuntu-12") in both the hostname command and
    /etc/hostname. Although that would be legal, the names were probably
    inconsistent somewhere. The Ubuntu installer stops you from violating some
    rules, such as ending a hostname with a dash (-), but it doesn't stop you
    from entering periods.

    So I take back my original claim: the minor version number change *does
    not* cause Hadoop to fail. The failure was due to one of the very few
    other differences between the VMs, the hostname.

    Thanks,
    -- Jim

    --
  • Harsh J at Oct 30, 2012 at 9:44 pm
    Thanks for following up Jim! So having fixed the hostnames, do your MR
    jobs run fine?

    One of my colleages, Clint, had filed an improvement to catch this
    environmental mistake better in logs, going forward:
    https://issues.apache.org/jira/browse/MAPREDUCE-4464, so that its not
    a pain anymore to debug it.

    On Tue, Oct 30, 2012 at 10:19 PM, Jim Pivarski
    wrote:
    On Tue, Oct 30, 2012 at 12:32 AM, Harsh J wrote:

    Do any of your hostnames in the cluster carry underscores in them
    and/or not-conform to RFC 1123?

    Yes, that seems to have been it, thank you!

    When I installed the distribution on the new VM, I entered a hostname that
    included periods (.) to indicate version number. The resulting VM was
    truncated at the position of the first period (that is, I wanted
    "ubuntu-12.04.1" but I got "ubuntu-12") in both the hostname command and
    /etc/hostname. Although that would be legal, the names were probably
    inconsistent somewhere. The Ubuntu installer stops you from violating some
    rules, such as ending a hostname with a dash (-), but it doesn't stop you
    from entering periods.

    So I take back my original claim: the minor version number change *does not*
    cause Hadoop to fail. The failure was due to one of the very few other
    differences between the VMs, the hostname.

    Thanks,
    -- Jim

    --



    --
    Harsh J

    --
  • Jim Pivarski at Oct 30, 2012 at 8:24 pm

    On Tue, Oct 30, 2012 at 12:59 PM, Harsh J wrote:

    Thanks for following up Jim! So having fixed the hostnames, do your MR
    jobs run fine?
    Yes--- I had tested the "grep" example before my last e-mail, but now I
    tested a simple streaming job in Python as well, and they both work
    (Java-based "grep" example and streaming).

    Moreover, both work on Ubuntu 12.10 also, if you use the Oracle JVM and the
    12.04.x CDH repositories:

    http://archive.cloudera.com/cdh4/one-click-install/*precise*
    /amd64/cdh4-repository_1.0_all.deb

    So at first glance, it seems like no changes are needed for the precise ->
    quantal transition.

    -- Jim

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedOct 30, '12 at 3:05a
activeOct 30, '12 at 9:44p
posts5
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Jim Pivarski: 3 posts Harsh J: 2 posts

People

Translate

site design / logo © 2018 Grokbase