FAQ
Hi guys,

i have following situation. A network setup with 2 network interfaces, eth0 (external) and eth1(internal).
Now in order to use the internal ips for hadoop i set dfs.datanode.dns.interface and mapred.tasktracker.dns.interface to eth1.
When tasktracker or datanode starting the following happens:
- hadoop picks the ip(s) from eth1
- hadoop tries to do a reverse dns lookup on the internal ip(s) - which fails
- hadoop fallback to InetAddress.getLocalHost() - which is the external dns

So in the end DNS.getDefaultIp(eth1) returns the internal ip whereas DNS.getDefaultHost(eth1) returns the external dns.

Isn't this a bug. Should in case the reverse lookup fails InetAddress.getByName(ips[ctr]).getCanonicalHost() rather be taken ?
Are there any suggested workarounds for this situation ?

best regards
Johannes

Search Discussions

  • Michael Segel at May 28, 2010 at 4:10 pm
    Hi,

    You can't do that.

    The problem is that Hadoop is going to pick up your external IP address because that's what the machine name resolves to. Then your slave nodes are on the internal route and you don't see them.

    Is it a bug? Maybe. More like a design defect.

    The work around is to forget about using the second nic card for hadoop/hbase traffic. Or make the internal network match your machine name and its dns information. Then use the second ip address to communicate with the outside world.

    So if your machine name is foo, and your dominiant ip address is on eth(0), you want foo.company.com to resolve to eth(0) and foo-ext.company.com
    to resolve to eth(1). Its backwards but it should work.

    IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.

    HTH

    -Mike

    PS. If I'm wrong please let me know. I had this problem and at the time, no solution was available.

    Subject: DNS.reverseDns() fallback
    From: jzillmann@googlemail.com
    Date: Fri, 28 May 2010 12:24:12 +0200
    To: core-user@hadoop.apache.org

    Hi guys,

    i have following situation. A network setup with 2 network interfaces, eth0 (external) and eth1(internal).
    Now in order to use the internal ips for hadoop i set dfs.datanode.dns.interface and mapred.tasktracker.dns.interface to eth1.
    When tasktracker or datanode starting the following happens:
    - hadoop picks the ip(s) from eth1
    - hadoop tries to do a reverse dns lookup on the internal ip(s) - which fails
    - hadoop fallback to InetAddress.getLocalHost() - which is the external dns

    So in the end DNS.getDefaultIp(eth1) returns the internal ip whereas DNS.getDefaultHost(eth1) returns the external dns.

    Isn't this a bug. Should in case the reverse lookup fails InetAddress.getByName(ips[ctr]).getCanonicalHost() rather be taken ?
    Are there any suggested workarounds for this situation ?

    best regards
    Johannes
    _________________________________________________________________
    Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
    http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1
  • Eric Sammer at May 28, 2010 at 5:14 pm

    On Fri, May 28, 2010 at 12:06 PM, Michael Segel wrote:

    Hi,

    You can't do that.
    Unfortunately, Mike is right.
    The problem is that Hadoop is going to pick up your external IP address because that's what the machine name resolves to. Then your slave nodes are on the internal route and you don't see them.

    Is it a bug? Maybe. More like a design defect.
    Definitely in the design defect category. The host name handling /
    binding code is... complicated and not ideal for these types of
    situations.
    The work around is to forget about using the second nic card for hadoop/hbase traffic. Or make the internal network match your machine name and its dns information. Then use the second ip address to communicate with the outside world.

    So if your machine name is foo, and your dominiant ip address is on eth(0), you want foo.company.com to resolve to eth(0) and foo-ext.company.com
    to resolve to eth(1). Its backwards but it should work.

    IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
    It might be useful in the case where you're streaming data off of HDFS
    directly to clients rather than in the MR or HBase case. Data import /
    export comes to mind. Remember that clients establish a direct
    connection to data nodes so a multihomed NN is insufficient. In that
    case, "external" doesn't necessarily mean a public (routable) IP, but
    simply another network. We've seen use cases for this in some
    installations. One example is a data aggregation or ingestion network
    is separate from the Hadoop internal network and you'd like to get
    data into HDFS.

    --
    Eric Sammer
    phone: +1-917-287-2675
    twitter: esammer
    data: www.cloudera.com
  • Michael Segel at May 28, 2010 at 5:47 pm

    Date: Fri, 28 May 2010 13:13:56 -0400
    Subject: Re: DNS.reverseDns() fallback
    From: esammer@cloudera.com
    To: common-user@hadoop.apache.org
    CC: core-user@hadoop.apache.org
    [SNIP]
    IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
    It might be useful in the case where you're streaming data off of HDFS
    directly to clients rather than in the MR or HBase case. Data import /
    export comes to mind. Remember that clients establish a direct
    connection to data nodes so a multihomed NN is insufficient. In that
    case, "external" doesn't necessarily mean a public (routable) IP, but
    simply another network. We've seen use cases for this in some
    installations. One example is a data aggregation or ingestion network
    is separate from the Hadoop internal network and you'd like to get
    data into HDFS.
    Eric,
    I don't disagree that it is useful.

    I was just offering up an alternative cloud design which would work as a work-around.
    As you point out, clients need to talk to all of the nodes.
    So you can make your name node your gateway for the rest of the cloud.
    (not a great idea but it works.)
    Or you can launch your jobs from a client or clients, who are multi-homed.

    If you're a Cloudera fan, you could use the cloudera desktop to launch jobs.
    So as long as that machine is visible to the cloud, you'll be OK too. ;-)

    I don't really recommend this, but its still a viable solution.

    -Mike



    _________________________________________________________________
    Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
    http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
  • Johannes Zillmann at Jun 1, 2010 at 11:49 am
    Thanks for your input!
    I added an issue for improving the fallback mechanism: https://issues.apache.org/jira/browse/HADOOP-6797

    Johannes
    On May 28, 2010, at 7:46 PM, Michael Segel wrote:



    Date: Fri, 28 May 2010 13:13:56 -0400
    Subject: Re: DNS.reverseDns() fallback
    From: esammer@cloudera.com
    To: common-user@hadoop.apache.org
    CC: core-user@hadoop.apache.org
    [SNIP]
    IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
    It might be useful in the case where you're streaming data off of HDFS
    directly to clients rather than in the MR or HBase case. Data import /
    export comes to mind. Remember that clients establish a direct
    connection to data nodes so a multihomed NN is insufficient. In that
    case, "external" doesn't necessarily mean a public (routable) IP, but
    simply another network. We've seen use cases for this in some
    installations. One example is a data aggregation or ingestion network
    is separate from the Hadoop internal network and you'd like to get
    data into HDFS.
    Eric,
    I don't disagree that it is useful.

    I was just offering up an alternative cloud design which would work as a work-around.
    As you point out, clients need to talk to all of the nodes.
    So you can make your name node your gateway for the rest of the cloud.
    (not a great idea but it works.)
    Or you can launch your jobs from a client or clients, who are multi-homed.

    If you're a Cloudera fan, you could use the cloudera desktop to launch jobs.
    So as long as that machine is visible to the cloud, you'll be OK too. ;-)

    I don't really recommend this, but its still a viable solution.

    -Mike



    _________________________________________________________________
    Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
    http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 28, '10 at 10:24a
activeJun 1, '10 at 11:49a
posts5
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase