FAQ
Hello,

I recently configured HA in CDH4 cluster. While trying to run oozie jobs I
get below error message

2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-oozie-oozi-W]
ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error starting
action [rtlsalejob]. ErrorType [ERROR], ErrorCode
[IllegalArgumentException], Message [IllegalArgumentException:
java.net.UnknownHostException: nameservice1]
org.apache.oozie.action.ActionExecutorException: IllegalArgumentException:
java.net.UnknownHostException: nameservice1
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
at org.apache.oozie.command.XCommand.call(XCommand.java:260)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalArgumentException:
java.net.UnknownHostException: nameservice1
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:328)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
at
org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
at
org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
at
org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at
org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
... 12 more
Caused by: java.net.UnknownHostException: nameservice1

Does anyone has idea what can be done? I tried changing the namenode in the
workflow.xml file to hdfs://nameservice1 but still not working?

Any help is appreciated.

Thanks
Hussain K

--

Search Discussions

  • Robert Kanter at Aug 9, 2012 at 4:07 pm
    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException). Perhaps
    there's something wrong with the network and nameservice1 isn't resolving
    properly. You can also try using the IP address instead of the hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie jobs I
    get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException: IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --


    --
  • Hussain k at Aug 9, 2012 at 4:41 pm
    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port as a
    name-node.
    With HA enabled we have to work with nameservice and not with the specific
    name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me the
    same error message. The Oozie workflow.xml has <name-node> tag but nothing
    like <nameservice> tag which is what we would expect in HA environment.

    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException). Perhaps
    there's something wrong with the network and nameservice1 isn't resolving
    properly. You can also try using the IP address instead of the hostname.

    Do either of those work?

    - Robert


    On Wed, Aug 8, 2012 at 10:46 AM, hussain k <iamhuss...@gmail.com<javascript:>
    wrote:
    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie jobs
    I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --


    --
  • Aaron T. Myers at Aug 9, 2012 at 5:29 pm
    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that includes
    all of the relevant client-side HA configs? In particular, the
    hdfs-site.xml should contain settings
    for dfs.nameservices, dfs.ha.namenodes.<ns
    ID>, dfs.namenode.rpc-address.<ns ID>.<nn ID>,
    and dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera


    On Thu, Aug 9, 2012 at 12:33 PM, hussain k wrote:

    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port as a
    name-node.
    With HA enabled we have to work with nameservice and not with the specific
    name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me the
    same error message. The Oozie workflow.xml has <name-node> tag but nothing
    like <nameservice> tag which is what we would expect in HA environment.

    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1 isn't
    resolving properly. You can also try using the IP address instead of the
    hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie jobs
    I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-**oozie-oozi-W]
    ACTION[0000010-**120802235149192-oozie-oozi-W@**rtlsalejob] Error
    starting action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.**ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at org.apache.oozie.action.**ActionExecutor.**convertException(**
    ActionExecutor.java:385)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    addToCache(JavaActionExecutor.**java:329)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    setLibFilesArchives(**JavaActionExecutor.java:413)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    submitLauncher(**JavaActionExecutor.java:592)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    start(JavaActionExecutor.java:**832)
    at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
    ActionStartXCommand.java:194)
    at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
    ActionStartXCommand.java:55)
    at org.apache.oozie.command.**XCommand.call(XCommand.java:**260)
    at org.apache.oozie.service.**CallableQueueService$**
    CompositeCallable.call(**CallableQueueService.java:300)
    at org.apache.oozie.service.**CallableQueueService$**
    CompositeCallable.call(**CallableQueueService.java:247)
    at org.apache.oozie.service.**CallableQueueService$**
    CallableWrapper.run(**CallableQueueService.java:166)
    at java.util.concurrent.**ThreadPoolExecutor$Worker.**
    runTask(ThreadPoolExecutor.**java:886)
    at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
    ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.**java:662)
    Caused by: java.lang.**IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at org.apache.hadoop.security.**SecurityUtil.**buildTokenService(**
    SecurityUtil.java:400)
    at org.apache.hadoop.hdfs.**NameNodeProxies.**createNonHAProxy(**
    NameNodeProxies.java:161)
    at org.apache.hadoop.hdfs.**NameNodeProxies.createProxy(**
    NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:364)
    at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:328)
    at org.apache.hadoop.hdfs.**DistributedFileSystem.**initialize(**
    DistributedFileSystem.java:**121)
    at org.apache.hadoop.fs.**FileSystem.createFileSystem(**
    FileSystem.java:2150)
    at org.apache.hadoop.fs.**FileSystem.access$200(**FileSystem.java:80)
    at org.apache.hadoop.fs.**FileSystem$Cache.getInternal(**
    FileSystem.java:2184)
    at org.apache.hadoop.fs.**FileSystem$Cache.get(**FileSystem.java:2166)
    at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:302)
    at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:158)
    at org.apache.hadoop.filecache.**DistributedCache.**addFileToClassPath(*
    *DistributedCache.java:431)
    at org.apache.oozie.service.**HadoopAccessorService$3.run(**
    HadoopAccessorService.java:**434)
    at org.apache.oozie.service.**HadoopAccessorService$3.run(**
    HadoopAccessorService.java:**429)
    at java.security.**AccessController.doPrivileged(**Native Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1232)
    at org.apache.oozie.service.**HadoopAccessorService.**
    addFileToClassPath(**HadoopAccessorService.java:**429)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    addToCache(JavaActionExecutor.**java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --


    --

    --
  • Hussain k at Aug 9, 2012 at 5:37 pm
    Hello Aaron,

    I have configured HA using Cloudera SCM manager and I assume it configures
    all the below parameters for me.Also I have downloaded client configuration
    URL from the SCM and verified the below setting in hdfs-site.xml.

    I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and see it
    has all the setting for HA. I have changed the server names in the below
    file to server.company.com for NDA reasons.

    Below is my hdfs-site.xml file

    <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
    <configuration>
       <property>
         <name>dfs.nameservices</name>
         <value>nameservice1</value>
       </property>
       <property>
         <name>dfs.client.failover.proxy.provider.nameservice1</name>

    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
       </property>
       <property>
         <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
         <value>true</value>
       </property>
       <property>
         <name>ha.zookeeper.quorum</name>
         <value>server.company.com:2181</value>
       </property>
       <property>
         <name>dfs.ha.namenodes.nameservice1</name>
         <value>namenode143,namenode73</value>
       </property>
       <property>
         <name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
         <value> server.company.com :8020</value>
       </property>
       <property>
         <name>dfs.namenode.http-address.nameservice1.namenode143</name>
         <value> server.company.com :50070</value>
       </property>
       <property>
         <name>dfs.namenode.https-address.nameservice1.namenode143</name>
         <value> server.company.com :50470</value>
       </property>
       <property>
         <name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
         <value> server.company.com :8020</value>
       </property>
       <property>
         <name>dfs.namenode.http-address.nameservice1.namenode73</name>
         <value> server.company.com :50070</value>
       </property>
       <property>
         <name>dfs.namenode.https-address.nameservice1.namenode73</name>
         <value> server.company.com :50470</value>
       </property>
       <property>
         <name>dfs.replication</name>
         <value>3</value>
       </property>
       <property>
         <name>dfs.blocksize</name>
         <value>134217728</value>
       </property>
       <property>
         <name>dfs.client.use.datanode.hostname</name>
         <value>false</value>
       </property>
    </configuration>


    Thanks for prompt response.

    Thanks
    Hussain K
    On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers wrote:

    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that includes
    all of the relevant client-side HA configs? In particular, the
    hdfs-site.xml should contain settings
    for dfs.nameservices, dfs.ha.namenodes.<ns
    ID>, dfs.namenode.rpc-address.<ns ID>.<nn ID>,
    and dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera


    On Thu, Aug 9, 2012 at 12:33 PM, hussain k wrote:

    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port as a
    name-node.
    With HA enabled we have to work with nameservice and not with the
    specific name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me the
    same error message. The Oozie workflow.xml has <name-node> tag but nothing
    like <nameservice> tag which is what we would expect in HA environment.

    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1 isn't
    resolving properly. You can also try using the IP address instead of the
    hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie
    jobs I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-**oozie-oozi-W]
    ACTION[0000010-**120802235149192-oozie-oozi-W@**rtlsalejob] Error
    starting action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.**ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at org.apache.oozie.action.**ActionExecutor.**convertException(**
    ActionExecutor.java:385)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    addToCache(JavaActionExecutor.**java:329)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    setLibFilesArchives(**JavaActionExecutor.java:413)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    submitLauncher(**JavaActionExecutor.java:592)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    start(JavaActionExecutor.java:**832)
    at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
    ActionStartXCommand.java:194)
    at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
    ActionStartXCommand.java:55)
    at org.apache.oozie.command.**XCommand.call(XCommand.java:**260)
    at org.apache.oozie.service.**CallableQueueService$**
    CompositeCallable.call(**CallableQueueService.java:300)
    at org.apache.oozie.service.**CallableQueueService$**
    CompositeCallable.call(**CallableQueueService.java:247)
    at org.apache.oozie.service.**CallableQueueService$**
    CallableWrapper.run(**CallableQueueService.java:166)
    at java.util.concurrent.**ThreadPoolExecutor$Worker.**
    runTask(ThreadPoolExecutor.**java:886)
    at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
    ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.**java:662)
    Caused by: java.lang.**IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at org.apache.hadoop.security.**SecurityUtil.**buildTokenService(**
    SecurityUtil.java:400)
    at org.apache.hadoop.hdfs.**NameNodeProxies.**createNonHAProxy(**
    NameNodeProxies.java:161)
    at org.apache.hadoop.hdfs.**NameNodeProxies.createProxy(**
    NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:364)
    at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:328)
    at org.apache.hadoop.hdfs.**DistributedFileSystem.**initialize(**
    DistributedFileSystem.java:**121)
    at org.apache.hadoop.fs.**FileSystem.createFileSystem(**
    FileSystem.java:2150)
    at org.apache.hadoop.fs.**FileSystem.access$200(**FileSystem.java:80)
    at org.apache.hadoop.fs.**FileSystem$Cache.getInternal(**
    FileSystem.java:2184)
    at org.apache.hadoop.fs.**FileSystem$Cache.get(**FileSystem.java:2166)
    at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:302)
    at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:158)
    at org.apache.hadoop.filecache.**DistributedCache.**addFileToClassPath(
    **DistributedCache.java:431)
    at org.apache.oozie.service.**HadoopAccessorService$3.run(**
    HadoopAccessorService.java:**434)
    at org.apache.oozie.service.**HadoopAccessorService$3.run(**
    HadoopAccessorService.java:**429)
    at java.security.**AccessController.doPrivileged(**Native Method)
    at javax.security.auth.Subject.**doAs(Subject.java:396)
    at org.apache.hadoop.security.**UserGroupInformation.doAs(**
    UserGroupInformation.java:**1232)
    at org.apache.oozie.service.**HadoopAccessorService.**
    addFileToClassPath(**HadoopAccessorService.java:**429)
    at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
    addToCache(JavaActionExecutor.**java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --


    --

    --




    --
    Regards,
    Hussain K

    --
  • Harsh J at Aug 10, 2012 at 12:56 pm
    Hi Hussain,

    We've identified a bug with some security related components that
    Hadoop uses, which isn't working well with HA configurations. We are
    looking into this presently and will send you more details shortly.
    On Thu, Aug 9, 2012 at 11:07 PM, hussain k wrote:
    Hello Aaron,

    I have configured HA using Cloudera SCM manager and I assume it configures
    all the below parameters for me.Also I have downloaded client configuration
    URL from the SCM and verified the below setting in hdfs-site.xml.

    I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and see it
    has all the setting for HA. I have changed the server names in the below
    file to server.company.com for NDA reasons.

    Below is my hdfs-site.xml file

    <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
    <configuration>
    <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
    </property>
    <property>
    <name>dfs.client.failover.proxy.provider.nameservice1</name>

    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
    <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
    <value>true</value>
    </property>
    <property>
    <name>ha.zookeeper.quorum</name>
    <value>server.company.com:2181</value>
    </property>
    <property>
    <name>dfs.ha.namenodes.nameservice1</name>
    <value>namenode143,namenode73</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode143</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode143</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode73</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode73</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.blocksize</name>
    <value>134217728</value>
    </property>
    <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>false</value>
    </property>
    </configuration>


    Thanks for prompt response.

    Thanks
    Hussain K
    On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers wrote:

    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that includes
    all of the relevant client-side HA configs? In particular, the hdfs-site.xml
    should contain settings for dfs.nameservices, dfs.ha.namenodes.<ns ID>,
    dfs.namenode.rpc-address.<ns ID>.<nn ID>, and
    dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Thu, Aug 9, 2012 at 12:33 PM, hussain k <iamhussain.110@gmail.com>
    wrote:
    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port as a
    name-node.
    With HA enabled we have to work with nameservice and not with the
    specific name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me the
    same error message. The Oozie workflow.xml has <name-node> tag but nothing
    like <nameservice> tag which is what we would expect in HA environment.

    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1 isn't
    resolving properly. You can also try using the IP address instead of the
    hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie
    jobs I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --

    --


    --




    --
    Regards,
    Hussain K


    --



    --
    Harsh J

    --
  • Harsh J at Aug 10, 2012 at 5:58 pm
    Hi Hussain,

    I think I was mistaken in my earlier response/analysis.

    Just to confirm somethings: Do all nodes of yours in the cluster
    include the HA configs (in /etc/hadoop/conf locations)? Since these
    actions run as tasks on a random node, you may require this (HA
    configs) to be present on all nodes for it to get picked up by them.
    On Fri, Aug 10, 2012 at 6:25 PM, Harsh J wrote:
    Hi Hussain,

    We've identified a bug with some security related components that
    Hadoop uses, which isn't working well with HA configurations. We are
    looking into this presently and will send you more details shortly.
    On Thu, Aug 9, 2012 at 11:07 PM, hussain k wrote:
    Hello Aaron,

    I have configured HA using Cloudera SCM manager and I assume it configures
    all the below parameters for me.Also I have downloaded client configuration
    URL from the SCM and verified the below setting in hdfs-site.xml.

    I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and see it
    has all the setting for HA. I have changed the server names in the below
    file to server.company.com for NDA reasons.

    Below is my hdfs-site.xml file

    <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
    <configuration>
    <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
    </property>
    <property>
    <name>dfs.client.failover.proxy.provider.nameservice1</name>

    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
    <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
    <value>true</value>
    </property>
    <property>
    <name>ha.zookeeper.quorum</name>
    <value>server.company.com:2181</value>
    </property>
    <property>
    <name>dfs.ha.namenodes.nameservice1</name>
    <value>namenode143,namenode73</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode143</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode143</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode73</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode73</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.blocksize</name>
    <value>134217728</value>
    </property>
    <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>false</value>
    </property>
    </configuration>


    Thanks for prompt response.

    Thanks
    Hussain K
    On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers wrote:

    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that includes
    all of the relevant client-side HA configs? In particular, the hdfs-site.xml
    should contain settings for dfs.nameservices, dfs.ha.namenodes.<ns ID>,
    dfs.namenode.rpc-address.<ns ID>.<nn ID>, and
    dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Thu, Aug 9, 2012 at 12:33 PM, hussain k <iamhussain.110@gmail.com>
    wrote:
    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port as a
    name-node.
    With HA enabled we have to work with nameservice and not with the
    specific name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me the
    same error message. The Oozie workflow.xml has <name-node> tag but nothing
    like <nameservice> tag which is what we would expect in HA environment.

    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1 isn't
    resolving properly. You can also try using the IP address instead of the
    hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie
    jobs I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException: nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --

    --


    --




    --
    Regards,
    Hussain K


    --



    --
    Harsh J


    --
    Harsh J

    --
  • Hussain k at Aug 10, 2012 at 6:48 pm
    Hello Harsh,

    All the nodes includes HA configs in /etc/hadoop/conf location.

    Regards
    Hussain K

    On Fri, Aug 10, 2012 at 12:50 PM, Harsh J wrote:

    Hi Hussain,

    I think I was mistaken in my earlier response/analysis.

    Just to confirm somethings: Do all nodes of yours in the cluster
    include the HA configs (in /etc/hadoop/conf locations)? Since these
    actions run as tasks on a random node, you may require this (HA
    configs) to be present on all nodes for it to get picked up by them.
    On Fri, Aug 10, 2012 at 6:25 PM, Harsh J wrote:
    Hi Hussain,

    We've identified a bug with some security related components that
    Hadoop uses, which isn't working well with HA configurations. We are
    looking into this presently and will send you more details shortly.
    On Thu, Aug 9, 2012 at 11:07 PM, hussain k wrote:
    Hello Aaron,

    I have configured HA using Cloudera SCM manager and I assume it
    configures
    all the below parameters for me.Also I have downloaded client
    configuration
    URL from the SCM and verified the below setting in hdfs-site.xml.

    I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and
    see it
    has all the setting for HA. I have changed the server names in the below
    file to server.company.com for NDA reasons.

    Below is my hdfs-site.xml file

    <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
    <configuration>
    <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
    </property>
    <property>
    <name>dfs.client.failover.proxy.provider.nameservice1</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
    <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
    <value>true</value>
    </property>
    <property>
    <name>ha.zookeeper.quorum</name>
    <value>server.company.com:2181</value>
    </property>
    <property>
    <name>dfs.ha.namenodes.nameservice1</name>
    <value>namenode143,namenode73</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode143</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode143</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode73</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode73</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.blocksize</name>
    <value>134217728</value>
    </property>
    <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>false</value>
    </property>
    </configuration>


    Thanks for prompt response.

    Thanks
    Hussain K
    On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers wrote:

    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that
    includes
    all of the relevant client-side HA configs? In particular, the
    hdfs-site.xml
    should contain settings for dfs.nameservices, dfs.ha.namenodes.<ns ID>,
    dfs.namenode.rpc-address.<ns ID>.<nn ID>, and
    dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Thu, Aug 9, 2012 at 12:33 PM, hussain k <iamhussain.110@gmail.com>
    wrote:
    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port
    as a
    name-node.
    With HA enabled we have to work with nameservice and not with the
    specific name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave me
    the
    same error message. The Oozie workflow.xml has <name-node> tag but
    nothing
    like <nameservice> tag which is what we would expect in HA
    environment.
    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1
    isn't
    resolving properly. You can also try using the IP address instead
    of the
    hostname.

    Do either of those work?

    - Robert

    On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie
    jobs I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf]
    JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error
    starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException:
    nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the
    namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --

    --


    --




    --
    Regards,
    Hussain K


    --



    --
    Harsh J


    --
    Harsh J

    --



    --
    Regards,
    Hussain K

    --
  • Hussain k at Aug 13, 2012 at 9:07 pm
    Any update as to what exactly is causing the error?

    Thanks
    Hussain k
    On Fri, Aug 10, 2012 at 1:48 PM, hussain k wrote:

    Hello Harsh,

    All the nodes includes HA configs in /etc/hadoop/conf location.

    Regards
    Hussain K

    On Fri, Aug 10, 2012 at 12:50 PM, Harsh J wrote:

    Hi Hussain,

    I think I was mistaken in my earlier response/analysis.

    Just to confirm somethings: Do all nodes of yours in the cluster
    include the HA configs (in /etc/hadoop/conf locations)? Since these
    actions run as tasks on a random node, you may require this (HA
    configs) to be present on all nodes for it to get picked up by them.
    On Fri, Aug 10, 2012 at 6:25 PM, Harsh J wrote:
    Hi Hussain,

    We've identified a bug with some security related components that
    Hadoop uses, which isn't working well with HA configurations. We are
    looking into this presently and will send you more details shortly.

    On Thu, Aug 9, 2012 at 11:07 PM, hussain k <iamhussain.110@gmail.com>
    wrote:
    Hello Aaron,

    I have configured HA using Cloudera SCM manager and I assume it
    configures
    all the below parameters for me.Also I have downloaded client
    configuration
    URL from the SCM and verified the below setting in hdfs-site.xml.

    I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and
    see it
    has all the setting for HA. I have changed the server names in the
    below
    file to server.company.com for NDA reasons.

    Below is my hdfs-site.xml file

    <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
    <configuration>
    <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
    </property>
    <property>
    <name>dfs.client.failover.proxy.provider.nameservice1</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
    <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
    <value>true</value>
    </property>
    <property>
    <name>ha.zookeeper.quorum</name>
    <value>server.company.com:2181</value>
    </property>
    <property>
    <name>dfs.ha.namenodes.nameservice1</name>
    <value>namenode143,namenode73</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode143</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode143</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
    <value> server.company.com :8020</value>
    </property>
    <property>
    <name>dfs.namenode.http-address.nameservice1.namenode73</name>
    <value> server.company.com :50070</value>
    </property>
    <property>
    <name>dfs.namenode.https-address.nameservice1.namenode73</name>
    <value> server.company.com :50470</value>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.blocksize</name>
    <value>134217728</value>
    </property>
    <property>
    <name>dfs.client.use.datanode.hostname</name>
    <value>false</value>
    </property>
    </configuration>


    Thanks for prompt response.

    Thanks
    Hussain K

    On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers <atm@cloudera.com>
    wrote:
    Are you positive that Sqoop/Oozie are using an hdfs-site.xml that
    includes
    all of the relevant client-side HA configs? In particular, the
    hdfs-site.xml
    should contain settings for dfs.nameservices, dfs.ha.namenodes.<ns
    ID>,
    dfs.namenode.rpc-address.<ns ID>.<nn ID>, and
    dfs.client.failover.proxy.provider.<ns ID>.

    --
    Aaron T. Myers
    Software Engineer, Cloudera



    On Thu, Aug 9, 2012 at 12:33 PM, hussain k <iamhussain.110@gmail.com>
    wrote:
    Hi Robert,

    This worked fine prior to HA.. when we specify specific server:port
    as a
    name-node.
    With HA enabled we have to work with nameservice and not with the
    specific name-node.

    By the way I already tried as hdfs://nameservice1:8020 and it gave
    me the
    same error message. The Oozie workflow.xml has <name-node> tag but
    nothing
    like <nameservice> tag which is what we would expect in HA
    environment.
    Let me know if you need any clarification

    Thanks for your time...

    Regards
    Hussain K


    On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

    Hi,

    I'm not sure when doing HA, but normally you need to include the
    port
    number with the name node in the workflow.xml. For example,
    hdfs://nameservice1:8020

    It looks like it can't find the host (i.e. UnknownHostException).
    Perhaps there's something wrong with the network and nameservice1
    isn't
    resolving properly. You can also try using the IP address instead
    of the
    hostname.

    Do either of those work?

    - Robert


    On Wed, Aug 8, 2012 at 10:46 AM, hussain k <iamhuss...@gmail.com>
    wrote:
    Hello,

    I recently configured HA in CDH4 cluster. While trying to run oozie
    jobs I get below error message

    2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
    GROUP[-] TOKEN[] APP[sqoop-wf]
    JOB[0000010-120802235149192-oozie-oozi-W]
    ACTION[0000010-120802235149192-oozie-oozi-W@rtlsalejob] Error
    starting
    action [rtlsalejob]. ErrorType [ERROR], ErrorCode
    [IllegalArgumentException], Message [IllegalArgumentException:
    java.net.UnknownHostException: nameservice1]
    org.apache.oozie.action.ActionExecutorException:
    IllegalArgumentException: java.net.UnknownHostException:
    nameservice1
    at
    org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:385)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:329)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:413)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:592)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:832)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:194)
    at
    org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
    at org.apache.oozie.command.XCommand.call(XCommand.java:260)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
    at
    org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
    at
    org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.IllegalArgumentException:
    java.net.UnknownHostException: nameservice1
    at
    org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:161)
    at
    org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:126)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:364)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:328)
    at
    org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:121)
    at
    org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2150)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
    at
    org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2184)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2166)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
    at
    org.apache.hadoop.filecache.DistributedCache.addFileToClassPath(DistributedCache.java:431)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:434)
    at
    org.apache.oozie.service.HadoopAccessorService$3.run(HadoopAccessorService.java:429)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at
    org.apache.oozie.service.HadoopAccessorService.addFileToClassPath(HadoopAccessorService.java:429)
    at
    org.apache.oozie.action.hadoop.JavaActionExecutor.addToCache(JavaActionExecutor.java:308)
    ... 12 more
    Caused by: java.net.UnknownHostException: nameservice1

    Does anyone has idea what can be done? I tried changing the
    namenode in
    the workflow.xml file to hdfs://nameservice1 but still not working?

    Any help is appreciated.

    Thanks
    Hussain K

    --

    --


    --




    --
    Regards,
    Hussain K


    --



    --
    Harsh J


    --
    Harsh J

    --



    --
    Regards,
    Hussain K


    --
    Regards,
    Hussain K

    --

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedAug 8, '12 at 5:54p
activeAug 13, '12 at 9:07p
posts9
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase