FAQ
I have a HDFS java client on the same network domain as the cluster which needs
to read the dfs file with permissions rw-r--r--

I get

java.io.IOException
at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:169)
at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)


Caused by: javax.security.auth.login.LoginException: Login failed: Expect one
token as the result of whoami: nt authority\\system
at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)

at
org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:167)\r\n\t... 20
more\r\n&msg=2010-08-06 16:25:38,069 ERROR -
com.ebay.integ.dal.cache2.CacheLoaderFailurenull\r\n


Can I use
job.set("dfs.permissions", "false");

My question is: Is it a matter of setting the right property values in the conf
object? like user.name or dfs.web.ugi or does the clinet need to actually have
a Unix login or passwordless ssh setup? -Ayon

Search Discussions

  • Josh Patterson at Aug 13, 2010 at 6:07 pm
    Ayon,
    When you have a java client that you want to use to simply access hdfs
    you need to make sure that you let your hdfs client know where to look
    to connect to hdfs and the namenode; You'll want to create a new
    Configuration object (which will automatically try and load a copy of
    your core-size.xml conf file if its around).

    conf = new Configuration();

    If you dont want to move a copy of it locally, you can manually set
    the parameter for the namenode:

    conf.set("fs.default.name", "hdfs://localhost:PORT");

    and then get a reference to hdfs.

    hdfs_ref = FileSystem.get(conf);

    from there, you can do basic FS operations like get a dir listing,
    create a file, delete a file, etc.

    Josh Patterson
    Cloudera
    On Mon, Aug 9, 2010 at 5:12 PM, Ayon Sinha wrote:
    I have a HDFS java client on the same network domain as the cluster which
    needs to read the dfs file with permissions  rw-r--r--
    I get
    java.io.IOException
    at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:169)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)

    Caused by: javax.security.auth.login.LoginException: Login failed: Expect
    one token as the result of whoami: nt authority\\system
    at
    org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
    at
    org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
    at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:167)\r\n\t... 20
    more\r\n&msg=2010-08-06 16:25:38,069 ERROR -
    com.ebay.integ.dal.cache2.CacheLoaderFailurenull\r\n

    Can I use
    job.set("dfs.permissions", "false");
    My question is: Is it a matter of setting the right property values in the
    conf object? like user.name or  dfs.web.ugi or does the clinet need to
    actually have a Unix login or passwordless ssh setup?

    -Ayon

  • Ayon Sinha at Aug 17, 2010 at 10:50 pm
    Hi,
    I have a java HDFS client which connects to a production cluster and gets data.
    On our staging environment we see that since it cannot connect to the namenode
    (expected) it keeps retrying for 45 times. I am looking for a way to set the
    retry count to much much lower.

    This is what we see in server logs:

    2010-08-17 15:15:06,973 INFO [Client] Retrying connect to server:
    xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 0 time(s).
    2010-08-17 15:15:27,979 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 1 time(s).
    2010-08-17 15:15:48,984 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 2 time(s).
    2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 3 time(s).
    ..
    ..
    ..
    2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 44 time(s).

    I have tried the client config
    job.set("ipc.client.connect.max.retries", "5");

    which doesn't seem to work.

    Where is this 45 reties coming from?

    Thanks in advance. -Ayon
  • Ayon Sinha at Aug 17, 2010 at 11:05 pm
    Looks like Client.java has the following code:

    } catch (SocketTimeoutException toe) {
    /* The max number of retries is 45,
    * which amounts to 20s*45 = 15 minutes retries.
    */
    handleConnectionFailure(timeoutFailures++, 45, toe);
    } catch (IOException ie) {
    handleConnectionFailure(ioFailures++, maxRetries, ie);
    }
    So it doesnt use the maxRetries variable when it gets a SocketTimeoutException.
    Our client is 0.18.3 (I know its ancient) but is this fixed in a later release?
    -Ayon






    ________________________________
    From: Ayon Sinha <ayonsinha@yahoo.com>
    To: hdfs-user@hadoop.apache.org
    Sent: Tue, August 17, 2010 3:49:56 PM
    Subject: HDFS java client connect retry count


    Hi,
    I have a java HDFS client which connects to a production cluster and gets data.
    On our staging environment we see that since it cannot connect to the namenode
    (expected) it keeps retrying for 45 times. I am looking for a way to set the
    retry count to much much lower.

    This is what we see in server logs:

    2010-08-17 15:15:06,973 INFO [Client] Retrying connect to server:
    xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 0 time(s).
    2010-08-17 15:15:27,979 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 1 time(s).
    2010-08-17 15:15:48,984 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 2 time(s).
    2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 3 time(s).
    ..
    ..
    ..
    2010-08-17 15:16:09,989 INFO [Client] Retrying connect to
    server: xxxx.yyyy.zzzz.com/192.168.1.11:9000. Already tried 44 time(s).

    I have tried the client config
    job.set("ipc.client.connect.max.retries", "5");

    which doesn't seem to work.

    Where is this 45 reties coming from?

    Thanks in advance. -Ayon

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouphdfs-user @
categorieshadoop
postedAug 9, '10 at 9:12p
activeAug 17, '10 at 11:05p
posts4
users2
websitehadoop.apache.org...
irc#hadoop

2 users in discussion

Ayon Sinha: 3 posts Josh Patterson: 1 post

People

Translate

site design / logo © 2022 Grokbase