Grokbase Groups HBase user July 2011
FAQ
Hi All,

I have a 5-node cluster setup with 3 nodes as a part of zookeeper quorum.
When i restart the hbase master, the server try to connect to an unknown
host and then crash.
Anyone seen this error message before or know how to resolve this thanks

2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: 10.16.129.21/10.16.129.21:50712
2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: 10.16.129.21/10.16.129.21:50712
2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
connecting to server: 10.16.129.21/10.16.129.21:50712
2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.net.SocketTimeoutException: 20000 millis timeout while waiting for
channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=
10.16.129.21/10.16.129.21:50712]
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get

Search Discussions

  • Buttler, David at Jul 15, 2011 at 4:54 pm
    You really don't need 3 zookeeper nodes for a 5 node cluster. 1 is sufficient.
    Are you managing zookeeper with hbase or independently?

    Dave


    -----Original Message-----
    From: Jason Chuong
    Sent: Friday, July 15, 2011 5:17 AM
    To: user@hbase.apache.org
    Subject: hbase crash after restart

    Hi All,

    I have a 5-node cluster setup with 3 nodes as a part of zookeeper quorum.
    When i restart the hbase master, the server try to connect to an unknown
    host and then crash.
    Anyone seen this error message before or know how to resolve this thanks

    2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
    Unhandled exception. Starting shutdown.
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for
    channel to be ready for connect. ch :
    java.nio.channels.SocketChannel[connection-pending remote=
    10.16.129.21/10.16.129.21:50712]
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy6.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get
  • Jason Chuong at Jul 15, 2011 at 5:59 pm
    Hi Dave,

    Yes we are and on hbase version 0.90, I've also verify that the zookeeper
    are responding via the zk shell and logs look normal.
    Just don't understand why it's trying to connect to that ip address.


    [zk: hadoop-wkr-r1:2181(CONNECTED) 1] ls /hbase
    [splitlog, unassigned, rs, root-region-server, table, shutdown]



    On Fri, Jul 15, 2011 at 9:54 AM, Buttler, David wrote:

    You really don't need 3 zookeeper nodes for a 5 node cluster. 1 is
    sufficient.
    Are you managing zookeeper with hbase or independently?

    Dave


    -----Original Message-----
    From: Jason Chuong
    Sent: Friday, July 15, 2011 5:17 AM
    To: user@hbase.apache.org
    Subject: hbase crash after restart

    Hi All,

    I have a 5-node cluster setup with 3 nodes as a part of zookeeper quorum.
    When i restart the hbase master, the server try to connect to an unknown
    host and then crash.
    Anyone seen this error message before or know how to resolve this thanks

    2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
    Unhandled exception. Starting shutdown.
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for
    channel to be ready for connect. ch :
    java.nio.channels.SocketChannel[connection-pending remote=
    10.16.129.21/10.16.129.21:50712]
    at

    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
    at

    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy6.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
    at

    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get
  • Bill Graham at Jul 15, 2011 at 6:21 pm
    What do you see when you do this from the ZK client:

    get /hbase/root-region-server

    I suspect a client somewhere registered itself in ZK. Maybe fixing the IP of
    the root region server in ZK will do the trick.

    On Fri, Jul 15, 2011 at 10:58 AM, Jason Chuong wrote:

    Hi Dave,

    Yes we are and on hbase version 0.90, I've also verify that the zookeeper
    are responding via the zk shell and logs look normal.
    Just don't understand why it's trying to connect to that ip address.


    [zk: hadoop-wkr-r1:2181(CONNECTED) 1] ls /hbase
    [splitlog, unassigned, rs, root-region-server, table, shutdown]



    On Fri, Jul 15, 2011 at 9:54 AM, Buttler, David wrote:

    You really don't need 3 zookeeper nodes for a 5 node cluster. 1 is
    sufficient.
    Are you managing zookeeper with hbase or independently?

    Dave


    -----Original Message-----
    From: Jason Chuong
    Sent: Friday, July 15, 2011 5:17 AM
    To: user@hbase.apache.org
    Subject: hbase crash after restart

    Hi All,

    I have a 5-node cluster setup with 3 nodes as a part of zookeeper quorum.
    When i restart the hbase master, the server try to connect to an unknown
    host and then crash.
    Anyone seen this error message before or know how to resolve this thanks

    2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
    Unhandled exception. Starting shutdown.
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for
    channel to be ready for connect. ch :
    java.nio.channels.SocketChannel[connection-pending remote=
    10.16.129.21/10.16.129.21:50712]
    at

    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
    at

    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy6.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
    at

    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get
  • Jason Chuong at Jul 15, 2011 at 9:51 pm
    that fix the problem bill, thanks for the help

    i when and remove the ip then restart the zookeeper port and everything came
    up.

    ie
    zk: hadoop-wkr1:2181(CONNECTED) 4] get /hbase/root-region-server
    10.16.129.21:50712
    cZxid = 77309767158
    ctime = Thu Jul 14 11:23:28 PDT 2011

    On Fri, Jul 15, 2011 at 11:21 AM, Bill Graham wrote:

    What do you see when you do this from the ZK client:

    get /hbase/root-region-server

    I suspect a client somewhere registered itself in ZK. Maybe fixing the IP
    of
    the root region server in ZK will do the trick.


    On Fri, Jul 15, 2011 at 10:58 AM, Jason Chuong <
    jason.chuong@cbsinteractive.com> wrote:
    Hi Dave,

    Yes we are and on hbase version 0.90, I've also verify that the zookeeper
    are responding via the zk shell and logs look normal.
    Just don't understand why it's trying to connect to that ip address.


    [zk: hadoop-wkr-r1:2181(CONNECTED) 1] ls /hbase
    [splitlog, unassigned, rs, root-region-server, table, shutdown]



    On Fri, Jul 15, 2011 at 9:54 AM, Buttler, David wrote:

    You really don't need 3 zookeeper nodes for a 5 node cluster. 1 is
    sufficient.
    Are you managing zookeeper with hbase or independently?

    Dave


    -----Original Message-----
    From: Jason Chuong
    Sent: Friday, July 15, 2011 5:17 AM
    To: user@hbase.apache.org
    Subject: hbase crash after restart

    Hi All,

    I have a 5-node cluster setup with 3 nodes as a part of zookeeper
    quorum.
    When i restart the hbase master, the server try to connect to an
    unknown
    host and then crash.
    Anyone seen this error message before or know how to resolve this
    thanks
    2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
    connecting to server: 10.16.129.21/10.16.129.21:50712
    2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
    Unhandled exception. Starting shutdown.
    java.net.SocketTimeoutException: 20000 millis timeout while waiting for
    channel to be ready for connect. ch :
    java.nio.channels.SocketChannel[connection-pending remote=
    10.16.129.21/10.16.129.21:50712]
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
    at
    org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
    at $Proxy6.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
    at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
    at
    org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshbase, hadoop
postedJul 15, '11 at 12:17p
activeJul 15, '11 at 9:51p
posts5
users3
websitehbase.apache.org

People

Translate

site design / logo © 2022 Grokbase