Slave node not able to send heart bits to master node.

On Sat, Nov 17, 2012 at 3:04 AM, Mehal Patel wrote:

I have configured master node as below:

file: /etc/hosts
added -> ipaddress_of_slave_node full_domain_name short_name

also changed host name in slave node's /etc/sysconfig/network file.
changed HOSTNAME to above mentioned full_domain_name.

Ports 7180 & 7182 are opened on Master nodes.

I am running CentOS 6.2 64 bit on virtual cloud servers and trying to
install CHD4 via cloudera manager

Please help me out.

Mehal

On Saturday, 17 November 2012 12:17:07 UTC-8, smark wrote:

Hi Mehal,

Could you paste the output of

# cat /etc/cloudera-scm-cat agent/config.ini | head -6

If you configured this by hand, ensure that the agent is configured to
heartbeat to the proper port (7182 by default) and not 7180.

Also, with /etc/hosts and /etc/sysconfig/network config, does

# hostname -f

return back the FQDN as expected?

If all the above looks good, paste a snippit of
/var/log/cloudera-scm-agent/cloudera-scm-agent.log from the node that
doesn't heartbeat as intended.

Regards,
--

On Sat, Nov 17, 2012 at 3:04 AM, Mehal Patel <[email protected]<javascript:>


--
*Mark Schnegelberger*

On Nov 19, 2012, at 1:48 AM, Mehal Patel wrote:

Hi Mark,

config.ini file has correct details ( 7182 port for master and 9000 on host itself ).
I am also able to see GOOD health in cloudera manager for that slave node but now i am getting this below error.

java.net.ConnectException: connection timed out to http://slave001.slave.com:9000/heartbeat
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:334)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:374)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: connection timed out
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
... 6 more

Any idea on this ?

On 19-Nov-2012, at 0:47, Dalia Hassan wrote:

Have you configured your dns server??

Sent from my iPad

On 2012-11-19, at 10:55 AM, Mehal wrote:

Hi Dalia,

Probably yes because i have cluster of 6 slaves and 1 master node and i am able to see them in cloudera manager with GOOD health. Only issue is when i see log file on master node it is showing error as i mentioned below.

Lwt me know if above makes sense.

Sent from my iPod

On Mon, Nov 19, 2012 at 8:58 AM, Dalia Hassan wrote:
Are all the services up Mehal ??

Sent from my iPhone

On 2012-11-19, at 10:55 AM, Mehal wrote:

Hi Dalia,

Probably yes because i have cluster of 6 slaves and 1 master node and i am
able to see them in cloudera manager with GOOD health. Only issue is when i
see log file on master node it is showing error as i mentioned below.

Lwt me know if above makes sense.

Sent from my iPod

On 19-Nov-2012, at 0:47, Dalia Hassan wrote:

Have you configured your dns server??

Sent from my iPad

On Nov 19, 2012, at 1:48 AM, Mehal Patel wrote:

Hi Mark,

config.ini file has correct details ( 7182 port for master and 9000 on host
itself ).
I am also able to see GOOD health in cloudera manager for that slave node
but now i am getting this below error.

java.net.ConnectException: connection timed out to
http://slave001.slave.com:9000/heartbeat
at
com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
at
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
at
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:334)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:374)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: connection timed out
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
... 6 more

Any idea on this ?

On Monday, November 19, 2012, Joey Echeverria wrote:

Can you make sure you don't have any software firewalls turned on?

-Joey
Joey Echeverria
Principal Solutions Architect
Cloudera, Inc.

On Mon, Nov 19, 2012 at 11:46 AM, Mehal Patel wrote:
I do have firewalls turned on but i have opened most of the ports in
/etc/sysconfig/iptables.

The thing is my slave as well master node us in domain slave.com ad
mentioned in the error.
So for each slave i have changed hostname inthe format such as
slave001.slave.com,slave002.slave.com etc through /etc/hosts and
/etc/sysconfig/network files with corresponding ipaddress field.

So i am able to access ports with ipaddress:port frim browser but not with
slave001.slave.com:port format through browser. So due to this is it getting
time out error?

Appreciate any help on this.

Mehal

On Monday, November 19, 2012, Joey Echeverria wrote:

You can check that DNS is set up correctly with the following commands:

dig slave001.slave.com
dig -x <ip for slave001.slave.com>

If one of those doesn't return the other, than DNS is not configured
correctly. If you're using /etc/hosts, make sure that it's configured
thusly:

127.0.0.1 localhost.localdomain
<ip of slave1> slave001.slave.com slave001
<ip of slave2> slave002.slave.com slave002
<...>

and make sure that the hostname command returns the FQDN:

$ hostname
slave001.slave.com

and not just the short name.

-Joey
i
com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:334)
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:374)
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

On Mon, Nov 19, 2012 at 1:13 PM, Mehal Patel wrote:
Hi Joey,

The below mentioned /etc/hosts should have details of master as well as all
slaves nodes on all the nodes in a cluster or only on the master node ?

And could you please brief upon dig command u mentioned? Did you mean that
both will turn out same results ? And does this command have to be issued on
master node only or can be issued on slave nodes also.

Thanks,
Mehal

On Monday, November 19, 2012, Joey Echeverria wrote:

Yes, the master should be in /etc/hosts as well.

The two dig commands won't return the exact same results, they should
basically return the input of the other. Can you run them on one of
your nodes and send the output?

-Joey
[email protected]>
and
is
below.
Joey Echeverria
Principal Solutions Architect
Cloudera, Inc.

On Monday, November 19, 2012, Dalia Hassan wrote:

Are all the services up Mehal ??

Sent from my iPhone

On 2012-11-19, at 10:55 AM, Mehal wrote:

Hi Dalia,

Probably yes because i have cluster of 6 slaves and 1 master node and i am
able to see them in cloudera manager with GOOD health. Only issue is when i
see log file on master node it is showing error as i mentioned below.

Lwt me know if above makes sense.

Sent from my iPod

On 19-Nov-2012, at 0:47, Dalia Hassan wrote:

Have you configured your dns server??

Sent from my iPad

On Nov 19, 2012, at 1:48 AM, Mehal Patel wrote:

Hi Mark,

config.ini file has correct details ( 7182 port for master and 9000 on
host itself ).
I am also able to see GOOD health in cloudera manager for that slave node
but now i am getting this below error.

*java.net.ConnectException: connection timed out to
http://slave001.slave.com:9000/heartbeat*
* at
com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
*
* at
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
*
* at
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
*
* at
org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:334)
*
* at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:374)
*
* at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:283)
*
* at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
*
* at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
*
* at java.lang.Thread.run(Thread.java:662)*
*Caused by: java.net.ConnectException: connection timed out*
* at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:371)
*
* ... 6 more*

Any idea on this ?

On Saturday, 17 November 2012 12:17:07 UTC-8, smark wrote:

Hi Mehal,

Could you paste the output of

# cat /etc/cloudera-scm-cat agent/config.ini | head -6

If you configured this by hand, ensure that the agent is configured to
heartbeat to the proper port (7182 by default) and not 7180.

Also, with /etc/hosts and /etc/sysconfig/network config, does

# hostname -f

return back the FQDN as expected?

If all the above looks good, paste a snippit of
/var/log/cloudera-scm-agent/**cloudera-scm-agent.log from the node that
doesn't heartbeat as intended.

Regards,
--

On Sat, Nov 17, 2012 at 3:04 AM, Mehal Patel wrote:

I have configured master node as below:

file: /etc/hosts
added -> ipaddress_of_slave_node full_domain_name short_name

also changed host name in slave node's /etc/sysconfig/network file.
changed HOSTNAME to above mentioned full_domain_name.

Ports 7180 & 7182 are opened on Master nodes.

I am running CentOS 6.2 64 bit on virtual cloud servers and trying to
install CHD4 via cloudera manager

Please help me out.

On Sat, Nov 17, 2012 at 12:16 PM, Mark Schnegelberger wrote:

Hi Mehal,

Could you paste the output of

# cat /etc/cloudera-scm-cat agent/config.ini | head -6

If you configured this by hand, ensure that the agent is configured to
heartbeat to the proper port (7182 by default) and not 7180.

Also, with /etc/hosts and /etc/sysconfig/network config, does

# hostname -f

return back the FQDN as expected?

If all the above looks good, paste a snippit of
/var/log/cloudera-scm-agent/cloudera-scm-agent.log from the node that
doesn't heartbeat as intended.

Regards,
--



--
*Mark Schnegelberger*