FAQ
I installed Cloudera Manager on Ubuntu 12.04 LTS successfuly. When I was in
the "Starting your cluster services.", the web page was stuck for a long
time in the step "Starting HDFS Service". When I tried to open
http://myserverip:7180/ it is no longer accessible.

I checked the logs in /var/log/cloudera-scm-server, I could not find errors.

I even tried restarting cloudera manager

root@emaillenin:/var/log/cloudera-scm-server# sudo service
cloudera-scm-server stop
Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed to
kill 19213: No such process
  * cloudera-scm-server stopped
root@emaillenin:/var/log/cloudera-scm-server# service cloudera-scm-server
start
Starting cloudera-scm-server: * cloudera-scm-server started

root@emaillenin:/var/log/cloudera-scm-server# netstat -p | grep 7180
root@emaillenin:/var/log/cloudera-scm-server#

And still I am not able to access the admin console at 7180

How should I debug this?

Search Discussions

  • Lenin Raj Rajasekaran at Jun 12, 2013 at 6:28 pm
    This is my /etc/hosts

    root@emaillenin:/var/log/hadoop-hdfs# cat /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 ubuntu
    96.126.114.83 emaillenin.com emaillenin

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    root@emaillenin:/var/log/hadoop-hdfs#


    When I checked the HDFS log here *
    /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-emaillenin.com.log.out:*

    Some of the errors handpicked from log:

    2013-06-12 23:00:22,910 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but
    service not yet registered with NN
    java.lang.Exception: trace
             at
    org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:154)
             at
    org.apache.hadoop.hdfs.server.datanode.DataNode.getNamenodeAddresses(DataNode.java:2320)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at
    com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
             at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
             at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
             at
    com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
             at
    com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
             at
    com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
             at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
             at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
             at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
             at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)



    2013-06-12 23:00:23,015 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting
    attribute VolumeInfo of Hadoop:service=DataNode,name=DataNodeInfo threw an
    exception
    javax.management.RuntimeMBeanException: java.lang.NullPointerException
             at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
             at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
             at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
             at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
             at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
             at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
             at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
             at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
             at
    org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1068)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
             at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
             at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
             at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
             at
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
             at
    org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)


    2013-06-12 23:16:46,934 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
    java.io.IOException: Failed on local exception: java.io.IOException:
    Response is null.; Host Details : local host is:
    "emaillenin.com/96.126.114.83"; destination host is: "emaillenin.com":8022;
             at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
             at org.apache.hadoop.ipc.Client.call(Client.java:1229)
             at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
             at $Proxy13.sendHeartbeat(Unknown Source)
             at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
             at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
             at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
             at $Proxy13.sendHeartbeat(Unknown Source)
             at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
             at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Response is null.
             at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:941)
             at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)


    2013-06-12 23:16:59,041 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
    java.net.ConnectException: *Call From emaillenin.com/96.126.114.83 to
    emaillenin.com:8022* failed on connection exception:
    java.net.ConnectException: Connection refused; For more details see:
    http://wiki.apache.org/hadoop/ConnectionRefused
             at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
             at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
             at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
             at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
             at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
             at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
             at org.apache.hadoop.ipc.Client.call(Client.java:1229)
             at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
             at $Proxy13.sendHeartbeat(Unknown Source)
             at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
             at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
             at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
             at $Proxy13.sendHeartbeat(Unknown Source)
             at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
             at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
             at java.lang.Thread.run(Thread.java:662)
    Caused by: java.net.ConnectException: Connection refused
             at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
             at
    sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
             at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
             at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
             at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)

    I am not sure why 8022 was being accessed and why Cloudera admin console is
    not starting up.

    On Wednesday, June 12, 2013 11:18:08 PM UTC+5:30, Lenin Raj Rajasekaran
    wrote:
    I installed Cloudera Manager on Ubuntu 12.04 LTS successfuly. When I was
    in the "Starting your cluster services.", the web page was stuck for a long
    time in the step "Starting HDFS Service". When I tried to open
    http://myserverip:7180/ it is no longer accessible.

    I checked the logs in /var/log/cloudera-scm-server, I could not find
    errors.

    I even tried restarting cloudera manager

    root@emaillenin:/var/log/cloudera-scm-server# sudo service
    cloudera-scm-server stop
    Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed to
    kill 19213: No such process
    * cloudera-scm-server stopped
    root@emaillenin:/var/log/cloudera-scm-server# service cloudera-scm-server
    start
    Starting cloudera-scm-server: * cloudera-scm-server started

    root@emaillenin:/var/log/cloudera-scm-server# netstat -p | grep 7180
    root@emaillenin:/var/log/cloudera-scm-server#

    And still I am not able to access the admin console at 7180

    How should I debug this?
  • bc Wong at Jun 12, 2013 at 6:30 pm
    Could you attach /var/log/cloudera-scm-server/cloudera-scm-server.log?
    (HDFS startup failure does not affect CM server.)

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 11:28 AM, Lenin Raj Rajasekaran
    wrote:
    This is my /etc/hosts

    root@emaillenin:/var/log/hadoop-hdfs# cat /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 ubuntu
    96.126.114.83 emaillenin.com emaillenin

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    root@emaillenin:/var/log/hadoop-hdfs#


    When I checked the HDFS log here
    /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-emaillenin.com.log.out:

    Some of the errors handpicked from log:

    2013-06-12 23:00:22,910 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but
    service not yet registered with NN
    java.lang.Exception: trace
    at
    org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:154)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.getNamenodeAddresses(DataNode.java:2320)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
    at
    com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    at
    com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
    at
    com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)



    2013-06-12 23:00:23,015 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting
    attribute VolumeInfo of Hadoop:service=DataNode,name=DataNodeInfo threw an
    exception
    javax.management.RuntimeMBeanException: java.lang.NullPointerException
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1068)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at
    org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)


    2013-06-12 23:16:46,934 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
    java.io.IOException: Failed on local exception: java.io.IOException:
    Response is null.; Host Details : local host is:
    "emaillenin.com/96.126.114.83"; destination host is: "emaillenin.com":8022;
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Response is null.
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:941)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)


    2013-06-12 23:16:59,041 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
    java.net.ConnectException: Call From emaillenin.com/96.126.114.83 to
    emaillenin.com:8022 failed on connection exception:
    java.net.ConnectException: Connection refused; For more details see:
    http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at
    sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)

    I am not sure why 8022 was being accessed and why Cloudera admin console is
    not starting up.


    On Wednesday, June 12, 2013 11:18:08 PM UTC+5:30, Lenin Raj Rajasekaran
    wrote:
    I installed Cloudera Manager on Ubuntu 12.04 LTS successfuly. When I was
    in the "Starting your cluster services.", the web page was stuck for a long
    time in the step "Starting HDFS Service". When I tried to open
    http://myserverip:7180/ it is no longer accessible.

    I checked the logs in /var/log/cloudera-scm-server, I could not find
    errors.

    I even tried restarting cloudera manager

    root@emaillenin:/var/log/cloudera-scm-server# sudo service
    cloudera-scm-server stop
    Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed to
    kill 19213: No such process
    * cloudera-scm-server stopped
    root@emaillenin:/var/log/cloudera-scm-server# service cloudera-scm-server
    start
    Starting cloudera-scm-server: * cloudera-scm-server started

    root@emaillenin:/var/log/cloudera-scm-server# netstat -p | grep 7180
    root@emaillenin:/var/log/cloudera-scm-server#

    And still I am not able to access the admin console at 7180

    How should I debug this?
  • Philip Zeyliger at Jun 12, 2013 at 6:32 pm
    Depending on your system, we've seen the Linux Out-of-Memory Killer choose
    us. You can see if it kicked in with "dmesg." Check to see if there's
    memory pressure on your system.

    -- Philip

    On Wed, Jun 12, 2013 at 11:30 AM, bc Wong wrote:

    Could you attach /var/log/cloudera-scm-server/cloudera-scm-server.log?
    (HDFS startup failure does not affect CM server.)

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 11:28 AM, Lenin Raj Rajasekaran
    wrote:
    This is my /etc/hosts

    root@emaillenin:/var/log/hadoop-hdfs# cat /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 ubuntu
    96.126.114.83 emaillenin.com emaillenin

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    root@emaillenin:/var/log/hadoop-hdfs#


    When I checked the HDFS log here
    /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-emaillenin.com.log.out:

    Some of the errors handpicked from log:

    2013-06-12 23:00:22,910 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but
    service not yet registered with NN
    java.lang.Exception: trace
    at
    org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:154)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.getNamenodeAddresses(DataNode.java:2320)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
    at
    com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    at
    com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
    at
    com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)


    2013-06-12 23:00:23,015 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting
    attribute VolumeInfo of Hadoop:service=DataNode,name=DataNodeInfo threw an
    exception
    javax.management.RuntimeMBeanException: java.lang.NullPointerException
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1068)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at
    org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    2013-06-12 23:16:46,934 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
    offerService
    java.io.IOException: Failed on local exception: java.io.IOException:
    Response is null.; Host Details : local host is:
    "emaillenin.com/96.126.114.83"; destination host is: "emaillenin.com ":8022;
    at
    org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Response is null.
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:941)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)


    2013-06-12 23:16:59,041 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
    offerService
    java.net.ConnectException: Call From emaillenin.com/96.126.114.83 to
    emaillenin.com:8022 failed on connection exception:
    java.net.ConnectException: Connection refused; For more details see:
    http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
    at
    org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at
    sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)

    I am not sure why 8022 was being accessed and why Cloudera admin console is
    not starting up.


    On Wednesday, June 12, 2013 11:18:08 PM UTC+5:30, Lenin Raj Rajasekaran
    wrote:
    I installed Cloudera Manager on Ubuntu 12.04 LTS successfuly. When I was
    in the "Starting your cluster services.", the web page was stuck for a
    long
    time in the step "Starting HDFS Service". When I tried to open
    http://myserverip:7180/ it is no longer accessible.

    I checked the logs in /var/log/cloudera-scm-server, I could not find
    errors.

    I even tried restarting cloudera manager

    root@emaillenin:/var/log/cloudera-scm-server# sudo service
    cloudera-scm-server stop
    Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed
    to
    kill 19213: No such process
    * cloudera-scm-server stopped
    root@emaillenin:/var/log/cloudera-scm-server# service
    cloudera-scm-server
    start
    Starting cloudera-scm-server: * cloudera-scm-server started

    root@emaillenin:/var/log/cloudera-scm-server# netstat -p | grep 7180
    root@emaillenin:/var/log/cloudera-scm-server#

    And still I am not able to access the admin console at 7180

    How should I debug this?
  • Lenin Raj at Jun 12, 2013 at 7:02 pm
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from yesterday).
    Another one has log from last 30 minutes.

    After my last email, I* restarted cloudera-scm-server*, the admin console *
    worked* at 7180. I saw the list of services such as hdfs, hive etc. I
    clicked *start* on HDFS service. It was showing progress but I lost
    connectivity to admin console again.

    *Hi Philip,*

    I do not see any error messages in dmesg. The file /var/log/dmesg was last
    edit 2 days back.

    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:02 AM, Philip Zeyliger wrote:

    Depending on your system, we've seen the Linux Out-of-Memory Killer choose
    us. You can see if it kicked in with "dmesg." Check to see if there's
    memory pressure on your system.

    -- Philip

    On Wed, Jun 12, 2013 at 11:30 AM, bc Wong wrote:

    Could you attach /var/log/cloudera-scm-server/cloudera-scm-server.log?
    (HDFS startup failure does not affect CM server.)

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 11:28 AM, Lenin Raj Rajasekaran
    wrote:
    This is my /etc/hosts

    root@emaillenin:/var/log/hadoop-hdfs# cat /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 ubuntu
    96.126.114.83 emaillenin.com emaillenin

    # The following lines are desirable for IPv6 capable hosts
    ::1 ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    root@emaillenin:/var/log/hadoop-hdfs#


    When I checked the HDFS log here
    /var/log/hadoop-hdfs/hadoop-cmf-hdfs1-DATANODE-emaillenin.com.log.out:

    Some of the errors handpicked from log:

    2013-06-12 23:00:22,910 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but
    service not yet registered with NN
    java.lang.Exception: trace
    at
    org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:154)
    at
    org.apache.hadoop.hdfs.server.datanode.DataNode.getNamenodeAddresses(DataNode.java:2320)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
    at
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
    at
    com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    at
    com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
    at
    com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)


    2013-06-12 23:00:23,015 ERROR org.apache.hadoop.jmx.JMXJsonServlet: getting
    attribute VolumeInfo of Hadoop:service=DataNode,name=DataNodeInfo threw an
    exception
    javax.management.RuntimeMBeanException: java.lang.NullPointerException
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
    at
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
    at
    com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:341)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:319)
    at
    org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:212)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at
    org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1068)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
    at
    org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at
    org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

    2013-06-12 23:16:46,934 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
    offerService
    java.io.IOException: Failed on local exception: java.io.IOException:
    Response is null.; Host Details : local host is:
    "emaillenin.com/96.126.114.83"; destination host is: "emaillenin.com ":8022;
    at
    org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.IOException: Response is null.
    at
    org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:941)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)


    2013-06-12 23:16:59,041 WARN
    org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
    offerService
    java.net.ConnectException: Call From emaillenin.com/96.126.114.83 to
    emaillenin.com:8022 failed on connection exception:
    java.net.ConnectException: Connection refused; For more details see:
    http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
    Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at
    java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
    at
    org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
    at org.apache.hadoop.ipc.Client.call(Client.java:1229)
    at
    org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
    at
    org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
    at $Proxy13.sendHeartbeat(Unknown Source)
    at
    org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:170)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:441)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:521)
    at
    org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    at java.lang.Thread.run(Thread.java:662)
    Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at
    sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)

    I am not sure why 8022 was being accessed and why Cloudera admin
    console is
    not starting up.


    On Wednesday, June 12, 2013 11:18:08 PM UTC+5:30, Lenin Raj Rajasekaran
    wrote:
    I installed Cloudera Manager on Ubuntu 12.04 LTS successfuly. When I
    was
    in the "Starting your cluster services.", the web page was stuck for a
    long
    time in the step "Starting HDFS Service". When I tried to open
    http://myserverip:7180/ it is no longer accessible.

    I checked the logs in /var/log/cloudera-scm-server, I could not find
    errors.

    I even tried restarting cloudera manager

    root@emaillenin:/var/log/cloudera-scm-server# sudo service
    cloudera-scm-server stop
    Stopping cloudera-scm-server: /sbin/start-stop-daemon: warning: failed
    to
    kill 19213: No such process
    * cloudera-scm-server stopped
    root@emaillenin:/var/log/cloudera-scm-server# service
    cloudera-scm-server
    start
    Starting cloudera-scm-server: * cloudera-scm-server started

    root@emaillenin:/var/log/cloudera-scm-server# netstat -p | grep 7180
    root@emaillenin:/var/log/cloudera-scm-server#

    And still I am not able to access the admin console at 7180

    How should I debug this?
  • bc Wong at Jun 12, 2013 at 7:15 pm

    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I clicked
    start on HDFS service. It was showing progress but I lost connectivity to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc
  • Lenin Raj at Jun 12, 2013 at 7:21 pm
    Hi Wong,

    I could not find messages in /var/log

    root@emaillenin:/var/log# ls mess*
    ls: cannot access mess*: No such file or directory

    Attached the kern.log. The last two lines are:

    Jun 13 00:18:48 tony kernel: Out of memory: Kill process 32353 (java)
    score 90 or sacrifice child
    Jun 13 00:18:48 tony kernel: Killed process 32353 (java)
    total-vm:909204kB, anon-rss:114212kB, file-rss:0kB

    I think the time coincides with the scm startup.

    What is the recommended RAM for running cloudera services? I have 1 GB
    of RAM. I would like to run it in just 1 node.





    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:45 AM, bc Wong wrote:
    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from
    yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I clicked
    start on HDFS service. It was showing progress but I lost connectivity to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc
  • Lenin Raj at Jun 12, 2013 at 7:45 pm
    Hi Wong,

    I am attaching the *htop *screenshot (sorted on Memory Percent).

    I see that there are multiple process being run for each services such as
    HDFS. Is this normal?



    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:51 AM, Lenin Raj wrote:

    Hi Wong,

    I could not find messages in /var/log

    root@emaillenin:/var/log# ls mess*
    ls: cannot access mess*: No such file or directory

    Attached the kern.log. The last two lines are:

    Jun 13 00:18:48 tony kernel: Out of memory: Kill process 32353 (java) score 90 or sacrifice child
    Jun 13 00:18:48 tony kernel: Killed process 32353 (java) total-vm:909204kB, anon-rss:114212kB, file-rss:0kB

    I think the time coincides with the scm startup.

    What is the recommended RAM for running cloudera services? I have 1 GB of RAM. I would like to run it in just 1 node.





    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:45 AM, bc Wong wrote:
    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from
    yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I clicked
    start on HDFS service. It was showing progress but I lost connectivity to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc
  • Marcelo Vanzin at Jun 12, 2013 at 7:51 pm

    On Wed, Jun 12, 2013 at 12:44 PM, Lenin Raj wrote:
    I am attaching the htop screenshot (sorted on Memory Percent).
    I see that there are multiple process being run for each services such as
    HDFS. Is this normal?
    Those are multiple threads of the same process. You can hit "H" inside
    htop to toggle that.

    --
    Marcelo
  • bc Wong at Jun 12, 2013 at 8:16 pm
    1 GB of RAM for CM + Hadoop is definitely not enough. Also, 1-node is
    not enough for most cases. If you're doing HDFS only (no MR), and you
    want to pack everything into 1 node, give yourself at least 4GB.

    If you want to try out CM + CDH, I'd recommend 4 nodes. Give the
    master node 8GB, and 4GB to the other 3.

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 12:21 PM, Lenin Raj wrote:
    Hi Wong,

    I could not find messages in /var/log

    root@emaillenin:/var/log# ls mess*
    ls: cannot access mess*: No such file or directory

    Attached the kern.log. The last two lines are:

    Jun 13 00:18:48 tony kernel: Out of memory: Kill process 32353 (java) score
    90 or sacrifice child
    Jun 13 00:18:48 tony kernel: Killed process 32353 (java) total-vm:909204kB,
    anon-rss:114212kB, file-rss:0kB

    I think the time coincides with the scm startup.

    What is the recommended RAM for running cloudera services? I have 1 GB of
    RAM. I would like to run it in just 1 node.






    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:45 AM, bc Wong wrote:
    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from
    yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I
    clicked
    start on HDFS service. It was showing progress but I lost connectivity
    to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was
    last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc
  • Lenin Raj at Jun 12, 2013 at 9:29 pm
    Hi Wong,

    Thanks for your suggestion.

    I am clueless about the sizing I need.

    I need to run about 20 Flume Agents - each pulling about 1000
    tweets/facebook posts per minute. And a MR job to find the top post in the
    past hour (based on retweets/share/like count). And display the top post in
    a site.

    I currently have about 20 GB free space in my server. This is enough for
    storing the raw data for a month, after which I can purge the month old
    data. The question is about the RAM. Are there any way I could manage to
    get this done within 1 GB RAM?


    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 1:46 AM, bc Wong wrote:

    1 GB of RAM for CM + Hadoop is definitely not enough. Also, 1-node is
    not enough for most cases. If you're doing HDFS only (no MR), and you
    want to pack everything into 1 node, give yourself at least 4GB.

    If you want to try out CM + CDH, I'd recommend 4 nodes. Give the
    master node 8GB, and 4GB to the other 3.

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 12:21 PM, Lenin Raj wrote:
    Hi Wong,

    I could not find messages in /var/log

    root@emaillenin:/var/log# ls mess*
    ls: cannot access mess*: No such file or directory

    Attached the kern.log. The last two lines are:

    Jun 13 00:18:48 tony kernel: Out of memory: Kill process 32353 (java) score
    90 or sacrifice child
    Jun 13 00:18:48 tony kernel: Killed process 32353 (java)
    total-vm:909204kB,
    anon-rss:114212kB, file-rss:0kB

    I think the time coincides with the scm startup.

    What is the recommended RAM for running cloudera services? I have 1 GB of
    RAM. I would like to run it in just 1 node.






    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:45 AM, bc Wong wrote:
    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from
    yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin
    console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I
    clicked
    start on HDFS service. It was showing progress but I lost connectivity
    to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was
    last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc
  • bc Wong at Jun 12, 2013 at 10:21 pm

    On Wed, Jun 12, 2013 at 2:29 PM, Lenin Raj wrote:
    Hi Wong,

    Thanks for your suggestion.

    I am clueless about the sizing I need.

    I need to run about 20 Flume Agents - each pulling about 1000
    tweets/facebook posts per minute. And a MR job to find the top post in the
    past hour (based on retweets/share/like count). And display the top post in
    a site.

    I currently have about 20 GB free space in my server. This is enough for
    storing the raw data for a month, after which I can purge the month old
    data. The question is about the RAM. Are there any way I could manage to get
    this done within 1 GB RAM?
    No way, not to fit HDFS, MR, Flume and CM in 1GB.

    To put us all in the right frame of mind regarding memory in 2013, my
    cellphone has more than 1GB of RAM.

    On Thu, Jun 13, 2013 at 1:46 AM, bc Wong wrote:

    1 GB of RAM for CM + Hadoop is definitely not enough. Also, 1-node is
    not enough for most cases. If you're doing HDFS only (no MR), and you
    want to pack everything into 1 node, give yourself at least 4GB.

    If you want to try out CM + CDH, I'd recommend 4 nodes. Give the
    master node 8GB, and 4GB to the other 3.

    Cheers,
    bc

    On Wed, Jun 12, 2013 at 12:21 PM, Lenin Raj wrote:
    Hi Wong,

    I could not find messages in /var/log

    root@emaillenin:/var/log# ls mess*
    ls: cannot access mess*: No such file or directory

    Attached the kern.log. The last two lines are:

    Jun 13 00:18:48 tony kernel: Out of memory: Kill process 32353 (java)
    score
    90 or sacrifice child
    Jun 13 00:18:48 tony kernel: Killed process 32353 (java)
    total-vm:909204kB,
    anon-rss:114212kB, file-rss:0kB

    I think the time coincides with the scm startup.

    What is the recommended RAM for running cloudera services? I have 1 GB
    of
    RAM. I would like to run it in just 1 node.






    Thanks,
    Lenin

    On Thu, Jun 13, 2013 at 12:45 AM, bc Wong wrote:

    On Wed, Jun 12, 2013 at 12:01 PM, Lenin Raj <emaillenin@gmail.com>
    wrote:
    Hi Wong,

    I have attached 2 log files. _old is a historic one (logs from
    yesterday).
    Another one has log from last 30 minutes.

    After my last email, I restarted cloudera-scm-server, the admin
    console
    worked at 7180. I saw the list of services such as hdfs, hive etc. I
    clicked
    start on HDFS service. It was showing progress but I lost
    connectivity
    to
    admin console again.
    Nothing in the CM server log. I'd bet money on Phil's guess.
    Hi Philip,

    I do not see any error messages in dmesg. The file /var/log/dmesg was
    last
    edit 2 days back.
    What about /var/log/messages, or /var/log/kern.log?

    Cheers,
    bc

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedJun 12, '13 at 5:48p
activeJun 12, '13 at 10:21p
posts12
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase