FAQ
Hello,
I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
Tha NameNode starts but the DataNode doesn't start and the log contains the following:

2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = clone1/147.102.4.129
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2-cdh3u0
STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
************************************************************/
2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

Does anyone know what might be wrong??

Thank you in advance!
Panagiotis

Search Discussions

  • Bharath Mundlapudi at May 13, 2011 at 1:27 am
    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG:  host = clone1/147.102.4.129
    STARTUP_MSG:  args = []
    STARTUP_MSG:  version = 0.20.2-cdh3u0
    STARTUP_MSG:  build =  -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis
  • Highpointe at May 13, 2011 at 5:59 am
    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = clone1/147.102.4.129
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis
  • Panayotis Antonopoulos at May 13, 2011 at 8:28 am
    There is no shutdown message until I shutdown the DataNode.

    I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference.
    Again the DataNode seems to freeze and the output at the log is the one I mentioned before.


    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Thu, 12 May 2011 23:59:02 -0600
    To: common-user@hadoop.apache.org

    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = clone1/147.102.4.129
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis
  • Highpointe at May 13, 2011 at 2:23 pm
    When you say "freeze" you mean there is nothing rolling in the log?

    Sent from my iPhone
    On May 13, 2011, at 2:28 AM, Panayotis Antonopoulos wrote:


    There is no shutdown message until I shutdown the DataNode.

    I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference.
    Again the DataNode seems to freeze and the output at the log is the one I mentioned before.


    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Thu, 12 May 2011 23:59:02 -0600
    To: common-user@hadoop.apache.org

    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = clone1/147.102.4.129
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis
  • Panayotis Antonopoulos at May 13, 2011 at 2:40 pm
    There is no other information in the log (although when I run it on my pc and it works, there is more information in the log) and also the web page of the namenode doesn't contain any live datanodes as it should.

    That's why I said it freezes... I have no idea what is going on...

    Please if anyone can help because it is really important to make it work as soon as possible (for my diploma thesis) and I really have no idea what might go wrong.

    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Fri, 13 May 2011 08:22:53 -0600
    To: common-user@hadoop.apache.org

    When you say "freeze" you mean there is nothing rolling in the log?

    Sent from my iPhone
    On May 13, 2011, at 2:28 AM, Panayotis Antonopoulos wrote:


    There is no shutdown message until I shutdown the DataNode.

    I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference.
    Again the DataNode seems to freeze and the output at the log is the one I mentioned before.


    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Thu, 12 May 2011 23:59:02 -0600
    To: common-user@hadoop.apache.org

    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = clone1/147.102.4.129
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis
  • Harsh J at May 13, 2011 at 4:29 pm
    Hello Panayotis,

    Could you please post a jstack output of your hung process to look into?

    $ jstack <PID of DN> # will do.

    2011/5/13 Panayotis Antonopoulos <antonopoulospan@hotmail.com>:
    There is no other information in the log (although when I run it on my pc and it works, there is more information in the log) and also the web page of the namenode doesn't contain any live datanodes as it should.

    That's why I said it freezes... I have no idea what is going on...

    Please if anyone can help because it is really important to make it work as soon as possible (for my diploma thesis) and I really have no idea what might go wrong.

    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Fri, 13 May 2011 08:22:53 -0600
    To: common-user@hadoop.apache.org

    When you say "freeze" you mean there is nothing rolling in the log?

    Sent from my iPhone
    On May 13, 2011, at 2:28 AM, Panayotis Antonopoulos wrote:


    There is no shutdown message until I shutdown the DataNode.

    I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference.
    Again the DataNode seems to freeze and the output at the log is the one I mentioned before.


    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Thu, 12 May 2011 23:59:02 -0600
    To: common-user@hadoop.apache.org

    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG:   host = clone1/147.102.4.129
    STARTUP_MSG:   args = []
    STARTUP_MSG:   version = 0.20.2-cdh3u0
    STARTUP_MSG:   build =  -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis


    --
    Harsh J
  • Panayotis Antonopoulos at May 13, 2011 at 5:36 pm
    Thank you for your help!
    Here is the result of the command that you said:

    panton@clone1:~/hadoop-0.20.203.0$ jstack 6320
    2011-05-13 20:31:59
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

    "Attach Listener" daemon prio=10 tid=0x00000000409c9800 nid=0x1999 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "pool-1-thread-1" prio=10 tid=0x00007f50f035f800 nid=0x1973 runnable [0x00007f50f6caf000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00000000eb403378> (a sun.nio.ch.Util$2)
    - locked <0x00000000eb403368> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000eb403160> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332)
    - locked <0x00000000eb403548> (a org.apache.hadoop.ipc.Server$Listener$Reader)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

    "Timer-0" daemon prio=10 tid=0x00007f50f84e6800 nid=0x195b in Object.wait() [0x00007f50f6db0000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec47a0a8> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x00000000ec47a0a8> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    "899599744@qtp-1416044437-1 - Acceptor0 SelectChannelConnector@0.0.0.0:50075" prio=10 tid=0x00007f50f8414800 nid=0x1926 runnable [0x00007f50f6eb1000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00000000ec473870> (a sun.nio.ch.Util$2)
    - locked <0x00000000ec473860> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000ec4733c8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

    "1620640756@qtp-1416044437-0" prio=10 tid=0x00007f50f83f0800 nid=0x1925 in Object.wait() [0x00007f50f75eb000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
    - locked <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

    "refreshUsed-/home/users/panton/hadoop-0.20.203.0/dfs/data" daemon prio=10 tid=0x00007f50f83f8000 nid=0x191f waiting on condition [0x00007f50f77fb000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    at java.lang.Thread.run(Thread.java:662)

    "Timer for 'DataNode' metrics system" daemon prio=10 tid=0x00007f50f83d3000 nid=0x18f5 in Object.wait() [0x00007f50f7b02000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec4407f8> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x00000000ec4407f8> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

    "RMI TCP Accept-0" daemon prio=10 tid=0x00007f50f835f800 nid=0x18e5 runnable [0x00007f50f7d04000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    - locked <0x00000000ec3e3930> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
    at java.lang.Thread.run(Thread.java:662)

    "Low Memory Detector" daemon prio=10 tid=0x00007f50f8025000 nid=0x18e1 runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread1" daemon prio=10 tid=0x00007f50f8022800 nid=0x18e0 waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "C2 CompilerThread0" daemon prio=10 tid=0x00007f50f801f800 nid=0x18df waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "Signal Dispatcher" daemon prio=10 tid=0x00007f50f801d800 nid=0x18de runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    "Finalizer" daemon prio=10 tid=0x00007f50f8001000 nid=0x18dd in Object.wait() [0x00007f50fc808000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec49c0f0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x00000000ec49c0f0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

    "Reference Handler" daemon prio=10 tid=0x0000000040181800 nid=0x18dc in Object.wait() [0x00007f50fc909000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec4923d8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x00000000ec4923d8> (a java.lang.ref.Reference$Lock)

    "main" prio=10 tid=0x0000000040115000 nid=0x18d2 runnable [0x00007f5101d2e000]
    java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:220)
    at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493)
    at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
    at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
    at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
    - locked <0x00000000eb430cc0> (a sun.security.provider.SecureRandom)
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - locked <0x00000000eb430f60> (a java.security.SecureRandom)
    at java.security.SecureRandom.next(SecureRandom.java:455)
    at java.util.Random.nextInt(Random.java:257)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573)

    "VM Thread" prio=10 tid=0x000000004017b000 nid=0x18db runnable

    "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040128000 nid=0x18d3 runnable

    "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004012a000 nid=0x18d4 runnable

    "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000004012c000 nid=0x18d5 runnable

    "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000004012e000 nid=0x18d6 runnable

    "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000004012f800 nid=0x18d7 runnable

    "GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040131800 nid=0x18d8 runnable

    "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000040133800 nid=0x18d9 runnable

    "GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000040135000 nid=0x18da runnable

    "VM Periodic Task Thread" prio=10 tid=0x00007f50f8374000 nid=0x18f2 waiting on condition

    JNI global references: 1741


    From: harsh@cloudera.com
    Date: Fri, 13 May 2011 21:58:41 +0530
    Subject: Re: Datanode doesn't start but there is no exception in the log
    To: common-user@hadoop.apache.org

    Hello Panayotis,

    Could you please post a jstack output of your hung process to look into?

    $ jstack <PID of DN> # will do.

    2011/5/13 Panayotis Antonopoulos <antonopoulospan@hotmail.com>:
    There is no other information in the log (although when I run it on my pc and it works, there is more information in the log) and also the web page of the namenode doesn't contain any live datanodes as it should.

    That's why I said it freezes... I have no idea what is going on...

    Please if anyone can help because it is really important to make it work as soon as possible (for my diploma thesis) and I really have no idea what might go wrong.

    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Fri, 13 May 2011 08:22:53 -0600
    To: common-user@hadoop.apache.org

    When you say "freeze" you mean there is nothing rolling in the log?

    Sent from my iPhone
    On May 13, 2011, at 2:28 AM, Panayotis Antonopoulos wrote:


    There is no shutdown message until I shutdown the DataNode.

    I used hostname of the machine that will run the DataNode and I now used the IP but there is no difference.
    Again the DataNode seems to freeze and the output at the log is the one I mentioned before.


    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: highpointe3i@gmail.com
    Date: Thu, 12 May 2011 23:59:02 -0600
    To: common-user@hadoop.apache.org

    Have you defined the IP
    of the DN in the slaves file?



    Sent from my iPhone
    On May 12, 2011, at 7:27 PM, Bharath Mundlapudi wrote:

    Is that all the messages in the datanode log? Do you see any SHUTDOWN message also?

    -Bharath



    ________________________________
    From: Panayotis Antonopoulos <antonopoulospan@hotmail.com>
    To: common-user@hadoop.apache.org
    Sent: Thursday, May 12, 2011 6:07 PM
    Subject: Datanode doesn't start but there is no exception in the log


    Hello,
    I am trying to set up Hadoop HDFS in a cluster for the first time. So far I was using pseudo-distributed mode on my PC at home and everything was working perfectly.
    Tha NameNode starts but the DataNode doesn't start and the log contains the following:

    2011-05-13 04:01:13,663 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
    /************************************************************
    STARTUP_MSG: Starting DataNode
    STARTUP_MSG: host = clone1/147.102.4.129
    STARTUP_MSG: args = []
    STARTUP_MSG: version = 0.20.2-cdh3u0
    STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'hudson' on Fri Mar 25 19:56:23 PDT 2011
    ************************************************************/
    2011-05-13 04:01:14,019 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
    2011-05-13 04:01:14,143 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
    2011-05-13 04:01:14,152 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
    2011-05-13 04:01:14,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
    2011-05-13 04:01:14,206 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    2011-05-13 04:01:14,272 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
    2011-05-13 04:01:14,278 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
    2011-05-13 04:01:14,278 INFO org.mortbay.log: jetty-6.1.26
    2011-05-13 04:01:14,567 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
    2011-05-13 04:01:14,570 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=DataNode, sessionId=null
    2011-05-13 04:01:14,976 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020
    2011-05-13 04:01:14,978 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,981 INFO org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics with hostName=DataNode, port=50020
    2011-05-13 04:01:14,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(clone1:50010, storageID=, infoPort=50075, ipcPort=50020)

    Does anyone know what might be wrong??

    Thank you in advance!
    Panagiotis


    --
    Harsh J
  • Harsh J at May 13, 2011 at 5:46 pm
    Hey,

    2011/5/13 Panayotis Antonopoulos <antonopoulospan@hotmail.com>:
    "899599744@qtp-1416044437-1 - Acceptor0 SelectChannelConnector@0.0.0.0:50075" prio=10 tid=0x00007f50f8414800 nid=0x1926 runnable [0x00007f50f6eb1000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00000000ec473870> (a sun.nio.ch.Util$2)
    - locked <0x00000000ec473860> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000ec4733c8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) and,
    "1620640756@qtp-1416044437-0" prio=10 tid=0x00007f50f83f0800 nid=0x1925 in Object.wait() [0x00007f50f75eb000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
    - locked <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread) and,
    "main" prio=10 tid=0x0000000040115000 nid=0x18d2 runnable [0x00007f5101d2e000]
    java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:220)
    at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493)
    at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
    at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
    at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
    - locked <0x00000000eb430cc0> (a sun.security.provider.SecureRandom)
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - locked <0x00000000eb430f60> (a java.security.SecureRandom)
    at java.security.SecureRandom.next(SecureRandom.java:455)
    at java.util.Random.nextInt(Random.java:257)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573)
    lead me to believe that you're running into this:
    http://search-hadoop.com/m/7Giae6vLWR1/securerandom&subj=Re+Entropy+Pool+and+HDFS+FS+Commands+Hanging+System

    Just give it some time and it should start up soon (you may generate
    some other activity on the DN to help it get some fresh entropy).
    Sometimes it may take up to a minute at start up.

    --
    Harsh J
  • Harsh J at May 13, 2011 at 5:49 pm
    Actually, only the last mentioned stack matters. Also see:
    https://issues.apache.org/jira/browse/HDFS-1835
    On Fri, May 13, 2011 at 11:15 PM, Harsh J wrote:
    Hey,

    2011/5/13 Panayotis Antonopoulos <antonopoulospan@hotmail.com>:
    "899599744@qtp-1416044437-1 - Acceptor0 SelectChannelConnector@0.0.0.0:50075" prio=10 tid=0x00007f50f8414800 nid=0x1926 runnable [0x00007f50f6eb1000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00000000ec473870> (a sun.nio.ch.Util$2)
    - locked <0x00000000ec473860> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000ec4733c8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) and,
    "1620640756@qtp-1416044437-0" prio=10 tid=0x00007f50f83f0800 nid=0x1925 in Object.wait() [0x00007f50f75eb000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
    - locked <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread) and,
    "main" prio=10 tid=0x0000000040115000 nid=0x18d2 runnable [0x00007f5101d2e000]
    java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:220)
    at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493)
    at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
    at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
    at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
    - locked <0x00000000eb430cc0> (a sun.security.provider.SecureRandom)
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - locked <0x00000000eb430f60> (a java.security.SecureRandom)
    at java.security.SecureRandom.next(SecureRandom.java:455)
    at java.util.Random.nextInt(Random.java:257)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573)
    lead me to believe that you're running into this:
    http://search-hadoop.com/m/7Giae6vLWR1/securerandom&subj=Re+Entropy+Pool+and+HDFS+FS+Commands+Hanging+System

    Just give it some time and it should start up soon (you may generate
    some other activity on the DN to help it get some fresh entropy).
    Sometimes it may take up to a minute at start up.

    --
    Harsh J


    --
    Harsh J
  • Panayotis Antonopoulos at May 13, 2011 at 6:19 pm
    I have been waiting for hours to see if it will ever start but it doesn't.
    I will check the links you sent me.

    Thanks again for your help!!!
    From: harsh@cloudera.com
    Date: Fri, 13 May 2011 23:18:40 +0530
    Subject: Re: Datanode doesn't start but there is no exception in the log
    To: common-user@hadoop.apache.org

    Actually, only the last mentioned stack matters. Also see:
    https://issues.apache.org/jira/browse/HDFS-1835
    On Fri, May 13, 2011 at 11:15 PM, Harsh J wrote:
    Hey,

    2011/5/13 Panayotis Antonopoulos <antonopoulospan@hotmail.com>:
    "899599744@qtp-1416044437-1 - Acceptor0 SelectChannelConnector@0.0.0.0:50075" prio=10 tid=0x00007f50f8414800 nid=0x1926 runnable [0x00007f50f6eb1000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x00000000ec473870> (a sun.nio.ch.Util$2)
    - locked <0x00000000ec473860> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000000ec4733c8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) and,
    "1620640756@qtp-1416044437-0" prio=10 tid=0x00007f50f83f0800 nid=0x1925 in Object.wait() [0x00007f50f75eb000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
    - locked <0x00000000ec473a88> (a org.mortbay.thread.QueuedThreadPool$PoolThread) and,
    "main" prio=10 tid=0x0000000040115000 nid=0x18d2 runnable [0x00007f5101d2e000]
    java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(Native Method)
    at java.io.FileInputStream.read(FileInputStream.java:220)
    at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedBytes(SeedGenerator.java:493)
    at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
    at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
    at sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
    - locked <0x00000000eb430cc0> (a sun.security.provider.SecureRandom)
    at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
    - locked <0x00000000eb430f60> (a java.security.SecureRandom)
    at java.security.SecureRandom.next(SecureRandom.java:455)
    at java.util.Random.nextInt(Random.java:257)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(DataNode.java:608)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.register(DataNode.java:629)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.runDatanodeDaemon(DataNode.java:1378)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1438)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1563)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1573)
    lead me to believe that you're running into this:
    http://search-hadoop.com/m/7Giae6vLWR1/securerandom&subj=Re+Entropy+Pool+and+HDFS+FS+Commands+Hanging+System

    Just give it some time and it should start up soon (you may generate
    some other activity on the DN to help it get some fresh entropy).
    Sometimes it may take up to a minute at start up.

    --
    Harsh J


    --
    Harsh J
  • Sridhar basam at May 13, 2011 at 7:10 pm
    Sounds like your entropy pool is exhausted blocking the process. What sort
    of hardware/os combo are you running this on?

    Sridhar
  • Panayotis Antonopoulos at May 14, 2011 at 12:49 am
    I am using the cluster of the Lab of my university, so I have limited access to its settings.
    It consists of 10 nodes with Intel Xeon CPUs and GNU/Linux 2.6.38

    Is there a way to solve the problem without changing the settings of the cluster?

    I am trying to use the patch that Harsh J sent me but I haven't managed to install it on hadoop properly yet.
    I have never installed a patch before...
    Date: Fri, 13 May 2011 15:09:38 -0400
    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: sri@basam.org
    To: common-user@hadoop.apache.org

    Sounds like your entropy pool is exhausted blocking the process. What sort
    of hardware/os combo are you running this on?

    Sridhar
  • Panayotis Antonopoulos at May 14, 2011 at 2:31 am
    I installed the patch:
    https://issues.apache.org/jira/browse/HDFS-1835
    that Harsh J told me and now everything works great!!!

    I hope that this change won't create other problems.

    Thanks to everyone and especially to Harsh J!!
    I would never find the problem without your help!!
    From: antonopoulospan@hotmail.com
    To: common-user@hadoop.apache.org
    Subject: RE: Datanode doesn't start but there is no exception in the log
    Date: Sat, 14 May 2011 03:49:21 +0300


    I am using the cluster of the Lab of my university, so I have limited access to its settings.
    It consists of 10 nodes with Intel Xeon CPUs and GNU/Linux 2.6.38

    Is there a way to solve the problem without changing the settings of the cluster?

    I am trying to use the patch that Harsh J sent me but I haven't managed to install it on hadoop properly yet.
    I have never installed a patch before...
    Date: Fri, 13 May 2011 15:09:38 -0400
    Subject: Re: Datanode doesn't start but there is no exception in the log
    From: sri@basam.org
    To: common-user@hadoop.apache.org

    Sounds like your entropy pool is exhausted blocking the process. What sort
    of hardware/os combo are you running this on?

    Sridhar

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedMay 13, '11 at 1:08a
activeMay 14, '11 at 2:31a
posts14
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase