FAQ
Hi all,

Are there anywhere instructions on how to change from the default ports
of Hadoop and HDFS? My main interest is in default port 8020.

Thanks,
George

--

Search Discussions

  • John Armstrong at Jun 2, 2011 at 2:32 pm

    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
  • George Kousiouris at Jun 2, 2011 at 2:49 pm
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall pov) port but still did not work. Maybe something else in
    addition is needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • Sulabh choudhury at Jun 2, 2011 at 3:57 pm
    Hi George,

    You have to set the HADOOP_SSH_OPTS in the *hadoop-env.sh*
    For eg :-
    export HADOOP_SSH_OPTS="-p 8221 -o ConnectTimeout=1 -o
    SendEnv=HADOOP_CONF_DIR"

    On Thu, Jun 2, 2011 at 7:48 AM, George Kousiouris wrote:


    Hi,

    thanks for the reply, we tried this with a definitely open (from a firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George

    On 6/2/2011 5:32 PM, John Armstrong wrote:

    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    --
    Thanks and Regards,
    Sulabh Choudhury
  • Harsh J at Jun 2, 2011 at 4:38 pm
    George,

    Could you additionally describe the issues you're facing right now w.r.t. ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/
    On Thu, Jun 2, 2011 at 8:18 PM, George Kousiouris wrote:

    Hi,

    thanks for the reply, we tried this with a definitely open (from a firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:

    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name.  You would go into core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece


    --
    Harsh J
  • George Kousiouris at Jun 2, 2011 at 4:48 pm
    Hi,

    the main issue that we are trying to connect a datanode from a remote
    domain to the master node that is behind a firewall that does not have
    8020 open. We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to have
    all this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now w.r.t. ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

    On Thu, Jun 2, 2011 at 8:18 PM, George Kousiouriswrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • Harsh J at Jun 2, 2011 at 4:57 pm
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a remote domain
    to the master node that is behind a firewall that does not have 8020 open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to have all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:

    George,

    Could you additionally describe the issues you're facing right now w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

    On Thu, Jun 2, 2011 at 8:18 PM, George Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:

    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name.  You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece


    --
    Harsh J
  • George Kousiouris at Jun 2, 2011 at 5:01 pm
    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a remote domain
    to the master node that is behind a firewall that does not have 8020 open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to have all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

    On Thu, Jun 2, 2011 at 8:18 PM, George Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • Le Zhao at Jun 2, 2011 at 6:02 pm
    btw, the /lezhao/gov2 service from the root directory
    (/bos/usr0/htdocs/lezhao/gov2) works fine.

    Le
    On 6/2/2011 1:00 PM, George Kousiouris wrote:

    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a remote
    domain
    to the master node that is behind a firewall that does not have 8020
    open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to
    have all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now
    w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/


    On Thu, Jun 2, 2011 at 8:18 PM, George
    Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default
    ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • George Kousiouris at Jun 2, 2011 at 6:09 pm
    Hi,

    we have made some progress with this. One of the problems was that:
    -etc/hosts master alias was set to localhost

    So we changed the core-site xml to have directly the public IP.

    BR,
    George
    On 6/2/2011 8:00 PM, George Kousiouris wrote:

    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a
    remote domain
    to the master node that is behind a firewall that does not have 8020
    open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to
    have all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now
    w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:
    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/


    On Thu, Jun 2, 2011 at 8:18 PM, George
    Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the
    default ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • Harsh J at Jun 2, 2011 at 6:13 pm
    George,

    Good to know. That'd indeed have lead to wrong bind/listen. Are you
    out of DFS troubles now?

    On Thu, Jun 2, 2011 at 11:39 PM, George Kousiouris
    wrote:
    Hi,

    we have made some progress with this. One of the problems was that:
    -etc/hosts master alias  was set to localhost

    So we changed the core-site xml to have directly the public IP.

    BR,
    George
    On 6/2/2011 8:00 PM, George Kousiouris wrote:

    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:

    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a remote
    domain
    to the master node that is behind a firewall that does not have 8020
    open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to have
    all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:

    George,

    Could you additionally describe the issues you're facing right now
    w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:

    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

    On Thu, Jun 2, 2011 at 8:18 PM, George
    Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:

    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default
    ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name.  You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece


    --
    Harsh J
  • George Kousiouris at Jun 2, 2011 at 6:31 pm
    Hi,

    well, sort of, we have some other networking issues now that forbid us
    from actually testing the change of the port.

    I will let you know as soon as all is tested.

    Thanks to everyone for their help,
    George
    On 6/2/2011 9:12 PM, Harsh J wrote:
    George,

    Good to know. That'd indeed have lead to wrong bind/listen. Are you
    out of DFS troubles now?

    On Thu, Jun 2, 2011 at 11:39 PM, George Kousiouris
    wrote:
    Hi,

    we have made some progress with this. One of the problems was that:
    -etc/hosts master alias was set to localhost

    So we changed the core-site xml to have directly the public IP.

    BR,
    George
    On 6/2/2011 8:00 PM, George Kousiouris wrote:
    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes, can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a remote
    domain
    to the master node that is behind a firewall that does not have 8020
    open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are started,
    datanode is listed in the jps, however in the datanode logs we get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful to have
    all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now
    w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post here
    you can get them from:

    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

    On Thu, Jun 2, 2011 at 8:18 PM, George
    Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the default
    ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
  • George Kousiouris at Jun 3, 2011 at 12:38 pm
    Hi all,

    with relation to this, we implemented another approach to work around
    the closed ports issues. We have setup a vpn between the namenode and
    the data nodes, so now everything is running ok.

    Thanks to everyone for your help,
    George

    On 6/2/2011 9:30 PM, George Kousiouris wrote:

    Hi,

    well, sort of, we have some other networking issues now that forbid us
    from actually testing the change of the port.

    I will let you know as soon as all is tested.

    Thanks to everyone for their help,
    George
    On 6/2/2011 9:12 PM, Harsh J wrote:
    George,

    Good to know. That'd indeed have lead to wrong bind/listen. Are you
    out of DFS troubles now?

    On Thu, Jun 2, 2011 at 11:39 PM, George Kousiouris
    wrote:
    Hi,

    we have made some progress with this. One of the problems was that:
    -etc/hosts master alias was set to localhost

    So we changed the core-site xml to have directly the public IP.

    BR,
    George
    On 6/2/2011 8:00 PM, George Kousiouris wrote:
    Hi,

    it seems to be working fine, and the telnet also works...:-(
    On 6/2/2011 7:56 PM, Harsh J wrote:
    George,

    Could you ensure that your 8090'd NameNode starts up fine? If yes,
    can
    also you telnet to that "host port" from your remote node - that
    should tell you if your firewall is at issue again?

    On Thu, Jun 2, 2011 at 10:17 PM, George Kousiouris
    wrote:
    Hi,

    the main issue that we are trying to connect a datanode from a
    remote
    domain
    to the master node that is behind a firewall that does not have 8020
    open.
    We change the port from the core-site xml to 8090 and restart.
    But when we try to start hdfs, it seems that all services are
    started,
    datanode is listed in the jps, however in the datanode logs we
    get the
    following error:

    2011-06-02 12:41:21,378 INFO org.apache.hadoop.ipc.Client: Retrying
    connect to server: master/xxx.xxx.xxx.xxx:8090. Already tried 3
    time(s).

    and it finally gives up. Thanks for the link, it is very helpful
    to have
    all
    this info concentrated.

    BR,
    George
    On 6/2/2011 7:37 PM, Harsh J wrote:
    George,

    Could you additionally describe the issues you're facing right now
    w.r.t.
    ports?

    If you need a fuller port number reference, we have a blog post
    here
    you can get them from:

    http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/


    On Thu, Jun 2, 2011 at 8:18 PM, George
    Kousiouris<gkousiou@mail.ntua.gr>
    wrote:
    Hi,

    thanks for the reply, we tried this with a definitely open (from a
    firewall
    pov) port but still did not work. Maybe something else in
    addition is
    needed?

    Thanks,
    George
    On 6/2/2011 5:32 PM, John Armstrong wrote:
    On Thu, 02 Jun 2011 17:23:08 +0300, George Kousiouris
    wrote:
    Are there anywhere instructions on how to change from the
    default
    ports
    of Hadoop and HDFS? My main interest is in default port 8020.
    I think this is part of fs.default.name. You would go into
    core-site.xml
    and add (or change)

    <property>
    <name>fs.default.name</name>
    <value>hdfs://[namenode]:[port]</value>
    </property>

    I'm sort of new to this myself, so if I'm wrong I hope someone
    will
    correct me.
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

    --

    ---------------------------

    George Kousiouris
    Electrical and Computer Engineer
    Division of Communications,
    Electronics and Information Engineering
    School of Electrical and Computer Engineering
    Tel: +30 210 772 2546
    Mobile: +30 6939354121
    Fax: +30 210 772 2569
    Email: gkousiou@mail.ntua.gr

    National Technical University of Athens
    9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJun 2, '11 at 2:23p
activeJun 3, '11 at 12:38p
posts13
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase