Can't access HDFS via Java code on HA for HDFS due to UnknownHostException: nameservice1

wrote:
Hi,


I am trying to do the following on *HDFS anf JobTracker High
Availability environment: *
1. *Access HDFS via java code*
My driver code:
public int run(String[] args) throws Exception {
...
final Configuration config= new Configuration(getConf());
config.set("mapred.job.tracker", "logicaljt"); //as defined in
mapred-site.xml
config.set(FileSystem.FS_DEFAULT_NAME_KEY,
"hdfs://nameservice1"); //as defined in core-site.xml
final FileSystem fs = FileSystem.get(config);
final FileStatus[] fileStatuses = fs.listStatus(new Path("/"));
// this line throws UnknownHostException: nameservice1
….
}
2. *Trigger mapreduce job via java code*
Similar to the code above, same initialization to Configuration object .

Both tests failed due to
java.lang.IllegalArgumentException: *java.net.UnknownHostException:
nameservice1*
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:448)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:410)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:127)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2273)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2307)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2289)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
at
com.peer39.hadooper.mr.aggregations.AggregationsDriver.run(AggregationsDriver.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.peer39.hadooper.daemon.AggregationDaemon.executeWork(AggregationDaemon.java:46)
at
com.peer39.commons.sandbox.daemon.scheduler.QuartzDaemon.execute(QuartzDaemon.java:147)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: java.net.UnknownHostException: nameservice1
... 19 more

- Both test run successfully on regular (not HA) environment.
- I've compared the my job configuration to another successful job
configuration (triggered by cmd, hadoop jar) and noticed *all special HA
configuration properties are missing in my job*…

My questions:
How to make it work? Is my Configuration object is well defined or there
are some missing properties? Should I specify all the special
configuration properties manually?
Is there something wrong with my HA configuration for HDFS (and Job
Tracker)?

I'm using CDH-4.7.2

Thanks,
Jasmin

--

---
You received this message because you are subscribed to the Google Groups
"CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to cdh-user+u...@cloudera.org <javascript:>.
For more options, visit
https://groups.google.com/a/cloudera.org/groups/opt_out.