oozie sqoop action error after adding High Availability

On Wed, Aug 8, 2012 at 10:46 AM, hussain k wrote:

Hello,

I recently configured HA in CDH4 cluster. While trying to run oozie jobs
I get below error message

2012-08-08 12:40:09,714 WARN ActionStartXCommand:542 - USER[hdfs]
GROUP[-] TOKEN[] APP[sqoop-wf] JOB[0000010-120802235149192-**oozie-oozi-W]
ACTION[0000010-**120802235149192-oozie-oozi-W@**rtlsalejob] Error
starting action [rtlsalejob]. ErrorType [ERROR], ErrorCode
[IllegalArgumentException], Message [IllegalArgumentException:
java.net.UnknownHostException: nameservice1]
org.apache.oozie.action.**ActionExecutorException:
IllegalArgumentException: java.net.UnknownHostException: nameservice1
at org.apache.oozie.action.**ActionExecutor.**convertException(**
ActionExecutor.java:385)
at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
addToCache(JavaActionExecutor.**java:329)
at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
setLibFilesArchives(**JavaActionExecutor.java:413)
at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
submitLauncher(**JavaActionExecutor.java:592)
at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
start(JavaActionExecutor.java:**832)
at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
ActionStartXCommand.java:194)
at org.apache.oozie.command.wf.**ActionStartXCommand.execute(**
ActionStartXCommand.java:55)
at org.apache.oozie.command.**XCommand.call(XCommand.java:**260)
at org.apache.oozie.service.**CallableQueueService$**
CompositeCallable.call(**CallableQueueService.java:300)
at org.apache.oozie.service.**CallableQueueService$**
CompositeCallable.call(**CallableQueueService.java:247)
at org.apache.oozie.service.**CallableQueueService$**
CallableWrapper.run(**CallableQueueService.java:166)
at java.util.concurrent.**ThreadPoolExecutor$Worker.**
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:662)
Caused by: java.lang.**IllegalArgumentException:
java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.**SecurityUtil.**buildTokenService(**
SecurityUtil.java:400)
at org.apache.hadoop.hdfs.**NameNodeProxies.**createNonHAProxy(**
NameNodeProxies.java:161)
at org.apache.hadoop.hdfs.**NameNodeProxies.createProxy(**
NameNodeProxies.java:126)
at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:364)
at org.apache.hadoop.hdfs.**DFSClient.<init>(DFSClient.**java:328)
at org.apache.hadoop.hdfs.**DistributedFileSystem.**initialize(**
DistributedFileSystem.java:**121)
at org.apache.hadoop.fs.**FileSystem.createFileSystem(**
FileSystem.java:2150)
at org.apache.hadoop.fs.**FileSystem.access$200(**FileSystem.java:80)
at org.apache.hadoop.fs.**FileSystem$Cache.getInternal(**
FileSystem.java:2184)
at org.apache.hadoop.fs.**FileSystem$Cache.get(**FileSystem.java:2166)
at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:302)
at org.apache.hadoop.fs.**FileSystem.get(FileSystem.**java:158)
at org.apache.hadoop.filecache.**DistributedCache.**addFileToClassPath(*
*DistributedCache.java:431)
at org.apache.oozie.service.**HadoopAccessorService$3.run(**
HadoopAccessorService.java:**434)
at org.apache.oozie.service.**HadoopAccessorService$3.run(**
HadoopAccessorService.java:**429)
at java.security.**AccessController.doPrivileged(**Native Method)
at javax.security.auth.Subject.**doAs(Subject.java:396)
at org.apache.hadoop.security.**UserGroupInformation.doAs(**
UserGroupInformation.java:**1232)
at org.apache.oozie.service.**HadoopAccessorService.**
addFileToClassPath(**HadoopAccessorService.java:**429)
at org.apache.oozie.action.**hadoop.JavaActionExecutor.**
addToCache(JavaActionExecutor.**java:308)
... 12 more
Caused by: java.net.UnknownHostException: nameservice1

Does anyone has idea what can be done? I tried changing the namenode in
the workflow.xml file to hdfs://nameservice1 but still not working?

Any help is appreciated.

Thanks
Hussain K

--

On Thursday, 9 August 2012 11:07:28 UTC-5, Robert Kanter wrote:

Hi,

I'm not sure when doing HA, but normally you need to include the port
number with the name node in the workflow.xml. For example,
hdfs://nameservice1:8020

It looks like it can't find the host (i.e. UnknownHostException).
Perhaps there's something wrong with the network and nameservice1 isn't
resolving properly. You can also try using the IP address instead of the
hostname.

Do either of those work?

- Robert

--

Hi Robert,

This worked fine prior to HA.. when we specify specific server:port as a
name-node.
With HA enabled we have to work with nameservice and not with the
specific name-node.

By the way I already tried as hdfs://nameservice1:8020 and it gave me the
same error message. The Oozie workflow.xml has <name-node> tag but nothing
like <nameservice> tag which is what we would expect in HA environment.

Let me know if you need any clarification

Thanks for your time...

Regards
Hussain K


--

On Thu, Aug 9, 2012 at 12:28 PM, Aaron T. Myers wrote:

Are you positive that Sqoop/Oozie are using an hdfs-site.xml that includes
all of the relevant client-side HA configs? In particular, the hdfs-site.xml
should contain settings for dfs.nameservices, dfs.ha.namenodes.<ns ID>,
dfs.namenode.rpc-address.<ns ID>.<nn ID>, and
dfs.client.failover.proxy.provider.<ns ID>.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Aug 9, 2012 at 12:33 PM, hussain k <iamhussain.110@gmail.com>
wrote:

--

On Thu, Aug 9, 2012 at 11:07 PM, hussain k wrote:
Hello Aaron,

I have configured HA using Cloudera SCM manager and I assume it

all the below parameters for me.Also I have downloaded client

URL from the SCM and verified the below setting in hdfs-site.xml.

I also looked into the /etc/hadoop/conf folder for hdfs-site.xml and

has all the setting for HA. I have changed the server names in the below
file to server.company.com for NDA reasons.

Below is my hdfs-site.xml file

<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera CM on 2012-08-08T17:21:53.995Z-->
<configuration>
<property>
<name>dfs.nameservices</name>
<value>nameservice1</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.nameservice1</name>

</property>
<property>
<name>dfs.ha.automatic-failover.enabled.nameservice1</name>
<value>true</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>server.company.com:2181</value>
</property>
<property>
<name>dfs.ha.namenodes.nameservice1</name>
<value>namenode143,namenode73</value>
</property>
<property>
<name>dfs.namenode.rpc-address.nameservice1.namenode143</name>
<value> server.company.com :8020</value>
</property>
<property>
<name>dfs.namenode.http-address.nameservice1.namenode143</name>
<value> server.company.com :50070</value>
</property>
<property>
<name>dfs.namenode.https-address.nameservice1.namenode143</name>
<value> server.company.com :50470</value>
</property>
<property>
<name>dfs.namenode.rpc-address.nameservice1.namenode73</name>
<value> server.company.com :8020</value>
</property>
<property>
<name>dfs.namenode.http-address.nameservice1.namenode73</name>
<value> server.company.com :50070</value>
</property>
<property>
<name>dfs.namenode.https-address.nameservice1.namenode73</name>
<value> server.company.com :50470</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>false</value>
</property>
</configuration>


Thanks for prompt response.

Thanks
Hussain K

--
Regards,
Hussain K


--

On Fri, Aug 10, 2012 at 6:25 PM, Harsh J wrote:
Hi Hussain,

We've identified a bug with some security related components that
Hadoop uses, which isn't working well with HA configurations. We are
looking into this presently and will send you more details shortly.

On Thu, Aug 9, 2012 at 11:07 PM, hussain k <iamhussain.110@gmail.com>

--
Harsh J