FAQ
Hi,

I've set up HDFS-HA with automatic failover, however, when I try to
run a MapReduce job, the following exception gets thrown:

13/02/20 15:56:33 INFO mapreduce.Export: verisons=1, starttime=0,
endtime=9223372036854775807
13/02/20 15:56:33 WARN conf.Configuration: dfs.df.interval is
deprecated. Instead, use fs.df.interval
13/02/20 15:56:33 WARN conf.Configuration: hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
13/02/20 15:56:33 WARN conf.Configuration: fs.default.name is
deprecated. Instead, use fs.defaultFS
13/02/20 15:56:33 WARN conf.Configuration: topology.script.number.args
is deprecated. Instead, use net.topology.script.number.args
13/02/20 15:56:33 WARN conf.Configuration:
topology.node.switch.mapping.impl is deprecated. Instead, use
net.topology.node.switch.mapping.impl
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.IllegalArgumentException:
java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:356)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:561)
at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
... 10 more
Caused by: java.net.UnknownHostException: nameservice1
... 38 more

"nameservice1" refers to the name of my HA NameNode Group. I have set
up Hbase to recognize this, by using
<property>
<name>hbase.rootdir</name>
<value>hdfs://nameservice1/hbase</value>
</property>
in hbase-site.xml, as recommended in the cloudera documentation.
However I am not quite sure how or where to set this, correctly for
MapReduce1. Some hints would be well appreciated!

Regards
Julian

P.S.:I am using Cloudera Manager 4.1.2 and CDH 4.1.2 in case this is
of any importance for resolving the issue.

Search Discussions

  • Vinithra Varadharajan at Feb 20, 2013 at 6:35 pm
    Julian,

    It looks like the client host from where you initiated the MR job does not
    have the full set of HA configs, particularly the one that maps the HA
    nameservice to the NameNodes. How did you go about setting up HDFS HA with
    CM? Did you use the "Enable High Availability" workflow? If you had, you
    would have had the option of deploying client configs, which you want to
    use - assuming that the client host has a CM agent running on it. The docs
    for enabling HDFS HA with CM are here:
    https://ccp.cloudera.com/display/FREE41DOC/Configuring+HDFS+High+Availability.
    With this, you won't need to modify hbase-site.xml or any other config file
    yourself.

    Hope this helps.

    -Vinithra

    On Wed, Feb 20, 2013 at 7:16 AM, Julian Wissmann
    wrote:
    Hi,

    I've set up HDFS-HA with automatic failover, however, when I try to
    run a MapReduce job, the following exception gets thrown:

    13/02/20 15:56:33 INFO mapreduce.Export: verisons=1, starttime=0,
    endtime=9223372036854775807
    13/02/20 15:56:33 WARN conf.Configuration: dfs.df.interval is
    deprecated. Instead, use fs.df.interval
    13/02/20 15:56:33 WARN conf.Configuration: hadoop.native.lib is
    deprecated. Instead, use io.native.lib.available
    13/02/20 15:56:33 WARN conf.Configuration: fs.default.name is
    deprecated. Instead, use fs.defaultFS
    13/02/20 15:56:33 WARN conf.Configuration: topology.script.number.args
    is deprecated. Instead, use net.topology.script.number.args
    13/02/20 15:56:33 WARN conf.Configuration:
    topology.node.switch.mapping.impl is deprecated. Instead, use
    net.topology.node.switch.mapping.impl
    Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
    at
    org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:561)
    at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:178)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at
    org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
    ... 10 more
    Caused by: java.net.UnknownHostException: nameservice1
    ... 38 more

    "nameservice1" refers to the name of my HA NameNode Group. I have set
    up Hbase to recognize this, by using
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://nameservice1/hbase</value>
    </property>
    in hbase-site.xml, as recommended in the cloudera documentation.
    However I am not quite sure how or where to set this, correctly for
    MapReduce1. Some hints would be well appreciated!

    Regards
    Julian

    P.S.:I am using Cloudera Manager 4.1.2 and CDH 4.1.2 in case this is
    of any importance for resolving the issue.
  • Vinithra Varadharajan at Feb 21, 2013 at 11:54 pm
    +scm-users

    Adding back scm-users so that others with the same problem see the end of
    this thread.

    Glad to know that deploying the client configs did the job!

    -Vinithra

    On Thu, Feb 21, 2013 at 3:25 PM, Julian Wissmann
    wrote:
    Hi,
    deploying the client config did, indeed help.
    Thank you!

    Julian

    2013/2/20 Vinithra Varadharajan <vinithra@cloudera.com>:
    Julian,

    It looks like the client host from where you initiated the MR job does not
    have the full set of HA configs, particularly the one that maps the HA
    nameservice to the NameNodes. How did you go about setting up HDFS HA with
    CM? Did you use the "Enable High Availability" workflow? If you had, you
    would have had the option of deploying client configs, which you want to use
    - assuming that the client host has a CM agent running on it. The docs for
    enabling HDFS HA with CM are here:
    https://ccp.cloudera.com/display/FREE41DOC/Configuring+HDFS+High+Availability
    .
    With this, you won't need to modify hbase-site.xml or any other config file
    yourself.

    Hope this helps.

    -Vinithra


    On Wed, Feb 20, 2013 at 7:16 AM, Julian Wissmann <
    julian.wissmann@sdace.de>
    wrote:
    Hi,

    I've set up HDFS-HA with automatic failover, however, when I try to
    run a MapReduce job, the following exception gets thrown:

    13/02/20 15:56:33 INFO mapreduce.Export: verisons=1, starttime=0,
    endtime=9223372036854775807
    13/02/20 15:56:33 WARN conf.Configuration: dfs.df.interval is
    deprecated. Instead, use fs.df.interval
    13/02/20 15:56:33 WARN conf.Configuration: hadoop.native.lib is
    deprecated. Instead, use io.native.lib.available
    13/02/20 15:56:33 WARN conf.Configuration: fs.default.name is
    deprecated. Instead, use fs.defaultFS
    13/02/20 15:56:33 WARN conf.Configuration: topology.script.number.args
    is deprecated. Instead, use net.topology.script.number.args
    13/02/20 15:56:33 WARN conf.Configuration:
    topology.node.switch.mapping.impl is deprecated. Instead, use
    net.topology.node.switch.mapping.impl
    Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:389)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:356)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:124)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2218)
    at
    org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2252)
    at
    org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2234)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:300)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
    at
    org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:902)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at
    org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:896)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:531)
    at
    org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:561)
    at
    org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:178)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at
    org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
    ... 10 more
    Caused by: java.net.UnknownHostException: nameservice1
    ... 38 more

    "nameservice1" refers to the name of my HA NameNode Group. I have set
    up Hbase to recognize this, by using
    <property>
    <name>hbase.rootdir</name>
    <value>hdfs://nameservice1/hbase</value>
    </property>
    in hbase-site.xml, as recommended in the cloudera documentation.
    However I am not quite sure how or where to set this, correctly for
    MapReduce1. Some hints would be well appreciated!

    Regards
    Julian

    P.S.:I am using Cloudera Manager 4.1.2 and CDH 4.1.2 in case this is
    of any importance for resolving the issue.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedFeb 20, '13 at 3:16p
activeFeb 21, '13 at 11:54p
posts3
users2
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase