FAQ
Hi All,

There is an issue with Namenode on Cloudera Manager. Whenever I start the
Namenode, It is not starting and giving this error message in log. The same
issue persist with HBase too.

Error:

Supervisor returned FATAL: + '[' -e
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
+ perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g'
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth
1 -name '*.py'
+ OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
+ '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
+ chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
+ export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
+ HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
+ export HADOOP_IDENT_STRING=hdfs
+ HADOOP_IDENT_STRING=hdfs
+ '[' -n '' ']'
+ acquire_kerberos_tgt hdfs.keytab
+ '[' -z hdfs.keytab ']'
+ '[' -n '' ']'
+ '[' validate-writable-empty-dirs = namenode ']'
+ '[' file-operation = namenode ']'
+ '[' bootstrap = namenode ']'
+ '[' failover = namenode ']'
+ '[' transition-to-active = namenode ']'
+ '[' initializeSharedEdits = namenode ']'
+ '[' initialize-znode = namenode ']'
+ '[' format-namenode = namenode ']'
+ '[' monitor-decommission = namenode ']'
+ '[' monitor-upgrade = namenode ']'
+ '[' finalize-upgrade = namenode ']'
+ '[' mkdir = namenode ']'
+ '[' namenode = namenode -o secondarynamenode = namenode -o datanode
= namenode ']'
+ HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS
-Djava.net.preferIPv4Stack=true '
+ export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT
-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
+ HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT
-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
+ exec /usr/lib/hadoop-hdfs/bin/hdfs --config
/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


Thanks & Regards,
*Anupam Ranjan*

Search Discussions

  • Darren Lo at Mar 19, 2013 at 3:30 pm
    Hi Anupam,
    (bcc cdh-user)

    Can you please provide any relevant role logs from Cloudera Manager?
    From the main page:
    Click HDFS
    Click on name node with problem
    Click Processes tab
    Click Role Log Details

    Please also provide logs for HBase.

    One common failure is a port conflict. If you see a message where it can't
    bind to a port, then check to make sure you don't have other processes
    running on those ports.

    Thanks,
    Darren

    On Tue, Mar 19, 2013 at 5:42 AM, Anupam Ranjan wrote:

    Hi All,

    There is an issue with Namenode on Cloudera Manager. Whenever I start the
    Namenode, It is not starting and giving this error message in log. The same
    issue persist with HBase too.

    Error:

    Supervisor returned FATAL: + '[' -e /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
    + perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g' /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    ++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth 1 -name '*.py'
    + OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
    + '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
    + chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
    + export HADOOP_IDENT_STRING=hdfs
    + HADOOP_IDENT_STRING=hdfs
    + '[' -n '' ']'
    + acquire_kerberos_tgt hdfs.keytab
    + '[' -z hdfs.keytab ']'
    + '[' -n '' ']'
    + '[' validate-writable-empty-dirs = namenode ']'
    + '[' file-operation = namenode ']'
    + '[' bootstrap = namenode ']'
    + '[' failover = namenode ']'
    + '[' transition-to-active = namenode ']'
    + '[' initializeSharedEdits = namenode ']'
    + '[' initialize-znode = namenode ']'
    + '[' format-namenode = namenode ']'
    + '[' monitor-decommission = namenode ']'
    + '[' monitor-upgrade = namenode ']'
    + '[' finalize-upgrade = namenode ']'
    + '[' mkdir = namenode ']'
    + '[' namenode = namenode -o secondarynamenode = namenode -o datanode = namenode ']'
    + HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + exec /usr/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


    Thanks & Regards,
    *Anupam Ranjan*


    --
    Thanks,
    Darren
  • Anupam Ranjan at Mar 20, 2013 at 4:58 am
    Hi Darren,

    PFA the role log files for Namenode and Hbase-master respectively.

    I don't see any other processes are running on this port.


    Thanks,
    *Anupam Ranjan*

    On 19 March 2013 21:00, Darren Lo wrote:

    Hi Anupam,
    (bcc cdh-user)

    Can you please provide any relevant role logs from Cloudera Manager?
    From the main page:
    Click HDFS
    Click on name node with problem
    Click Processes tab
    Click Role Log Details

    Please also provide logs for HBase.

    One common failure is a port conflict. If you see a message where it can't
    bind to a port, then check to make sure you don't have other processes
    running on those ports.

    Thanks,
    Darren

    On Tue, Mar 19, 2013 at 5:42 AM, Anupam Ranjan wrote:

    Hi All,

    There is an issue with Namenode on Cloudera Manager. Whenever I start the
    Namenode, It is not starting and giving this error message in log. The same
    issue persist with HBase too.

    Error:

    Supervisor returned FATAL: + '[' -e /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
    + perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g' /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    ++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth 1 -name '*.py'
    + OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
    + '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
    + chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
    + export HADOOP_IDENT_STRING=hdfs
    + HADOOP_IDENT_STRING=hdfs
    + '[' -n '' ']'
    + acquire_kerberos_tgt hdfs.keytab
    + '[' -z hdfs.keytab ']'
    + '[' -n '' ']'
    + '[' validate-writable-empty-dirs = namenode ']'
    + '[' file-operation = namenode ']'
    + '[' bootstrap = namenode ']'
    + '[' failover = namenode ']'
    + '[' transition-to-active = namenode ']'
    + '[' initializeSharedEdits = namenode ']'
    + '[' initialize-znode = namenode ']'
    + '[' format-namenode = namenode ']'
    + '[' monitor-decommission = namenode ']'
    + '[' monitor-upgrade = namenode ']'
    + '[' finalize-upgrade = namenode ']'
    + '[' mkdir = namenode ']'
    + '[' namenode = namenode -o secondarynamenode = namenode -o datanode = namenode ']'
    + HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + exec /usr/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


    Thanks & Regards,
    *Anupam Ranjan*


    --
    Thanks,
    Darren
  • Darren Lo at Mar 20, 2013 at 4:25 pm
    (Adding back cdh-user, seems to be a cdh issue)

    Seems there was a problem with recovering your namenode transactions. Were
    there any crashes, abnormal restarts, hard drive problems, etc recently?

    Log tail:
    13-03-20 10:25:53,620 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /dfs/nn/in_use.lock acquired by nodename 11230@clouderra.tcubes.com
    2013-03-20 10:25:53,731 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /data/dfs/nn/in_use.lock acquired by nodename
    11230@clouderra.tcubes.com
    2013-03-20 10:25:53,880 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /dfs/nn/current
    2013-03-20 10:25:53,988 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /data/dfs/nn/current
    2013-03-20 10:25:54,581 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
    metrics system...
    2013-03-20 10:25:54,582 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    stopped.
    2013-03-20 10:25:54,582 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    shutdown complete.
    2013-03-20 10:25:54,583 FATAL
    org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
    java.io.IOException: Gap in transactions. Expected to be able to read up
    until at least txid 2647 but unable to find any edit logs containing txid
    2647
    at
    org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1175)
    at
    org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1133)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:616)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:267)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:589)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
    2013-03-20 10:25:54,592 INFO org.apache.hadoop.util.ExitUtil: Exiting with
    status 1
    2013-03-20 10:25:54,600 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at clouderra.tcubes.com/192.168.3.227
    ************************************************************/

    Hbase can't start because of a connection error, probably because your
    namenode is down.

    Thanks,
    Darren

    On Tue, Mar 19, 2013 at 9:58 PM, Anupam Ranjan wrote:

    Hi Darren,

    PFA the role log files for Namenode and Hbase-master respectively.

    I don't see any other processes are running on this port.


    Thanks,
    *Anupam Ranjan*

    On 19 March 2013 21:00, Darren Lo wrote:

    Hi Anupam,
    (bcc cdh-user)

    Can you please provide any relevant role logs from Cloudera Manager?
    From the main page:
    Click HDFS
    Click on name node with problem
    Click Processes tab
    Click Role Log Details

    Please also provide logs for HBase.

    One common failure is a port conflict. If you see a message where it
    can't bind to a port, then check to make sure you don't have other
    processes running on those ports.

    Thanks,
    Darren


    On Tue, Mar 19, 2013 at 5:42 AM, Anupam Ranjan <fantasticanupam@gmail.com
    wrote:
    Hi All,

    There is an issue with Namenode on Cloudera Manager. Whenever I start
    the Namenode, It is not starting and giving this error message in log. The
    same issue persist with HBase too.

    Error:

    Supervisor returned FATAL: + '[' -e /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
    + perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g' /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    ++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth 1 -name '*.py'
    + OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
    + '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
    + chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
    + export HADOOP_IDENT_STRING=hdfs
    + HADOOP_IDENT_STRING=hdfs
    + '[' -n '' ']'
    + acquire_kerberos_tgt hdfs.keytab
    + '[' -z hdfs.keytab ']'
    + '[' -n '' ']'
    + '[' validate-writable-empty-dirs = namenode ']'
    + '[' file-operation = namenode ']'
    + '[' bootstrap = namenode ']'
    + '[' failover = namenode ']'
    + '[' transition-to-active = namenode ']'
    + '[' initializeSharedEdits = namenode ']'
    + '[' initialize-znode = namenode ']'
    + '[' format-namenode = namenode ']'
    + '[' monitor-decommission = namenode ']'
    + '[' monitor-upgrade = namenode ']'
    + '[' finalize-upgrade = namenode ']'
    + '[' mkdir = namenode ']'
    + '[' namenode = namenode -o secondarynamenode = namenode -o datanode = namenode ']'
    + HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + exec /usr/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


    Thanks & Regards,
    *Anupam Ranjan*


    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • Anupam Ranjan at Mar 22, 2013 at 12:49 pm
    Thanks Darren,

    But there is non of this kind of things happened recently.

    is there any solution for this or How I can recover from this issue?

    Every time I start or restart, I am the same issue.


    Thanks & Regards,

    *Anupam Ranjan*

    On 20 March 2013 21:55, Darren Lo wrote:

    (Adding back cdh-user, seems to be a cdh issue)

    Seems there was a problem with recovering your namenode transactions. Were
    there any crashes, abnormal restarts, hard drive problems, etc recently?

    Log tail:
    13-03-20 10:25:53,620 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /dfs/nn/in_use.lock acquired by nodename
    11230@clouderra.tcubes.com
    2013-03-20 10:25:53,731 INFO org.apache.hadoop.hdfs.server.common.Storage:
    Lock on /data/dfs/nn/in_use.lock acquired by nodename
    11230@clouderra.tcubes.com
    2013-03-20 10:25:53,880 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /dfs/nn/current
    2013-03-20 10:25:53,988 INFO
    org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
    unfinalized segments in /data/dfs/nn/current
    2013-03-20 10:25:54,581 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
    metrics system...
    2013-03-20 10:25:54,582 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    stopped.
    2013-03-20 10:25:54,582 INFO
    org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
    shutdown complete.
    2013-03-20 10:25:54,583 FATAL
    org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
    java.io.IOException: Gap in transactions. Expected to be able to read up
    until at least txid 2647 but unable to find any edit logs containing txid
    2647
    at
    org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1175)
    at
    org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1133)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:616)
    at
    org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:267)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
    at
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
    at
    org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
    2013-03-20 10:25:54,592 INFO org.apache.hadoop.util.ExitUtil: Exiting with
    status 1
    2013-03-20 10:25:54,600 INFO
    org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at clouderra.tcubes.com/192.168.3.227
    ************************************************************/

    Hbase can't start because of a connection error, probably because your
    namenode is down.

    Thanks,
    Darren

    On Tue, Mar 19, 2013 at 9:58 PM, Anupam Ranjan wrote:

    Hi Darren,

    PFA the role log files for Namenode and Hbase-master respectively.

    I don't see any other processes are running on this port.


    Thanks,
    *Anupam Ranjan*

    On 19 March 2013 21:00, Darren Lo wrote:

    Hi Anupam,
    (bcc cdh-user)

    Can you please provide any relevant role logs from Cloudera Manager?
    From the main page:
    Click HDFS
    Click on name node with problem
    Click Processes tab
    Click Role Log Details

    Please also provide logs for HBase.

    One common failure is a port conflict. If you see a message where it
    can't bind to a port, then check to make sure you don't have other
    processes running on those ports.

    Thanks,
    Darren


    On Tue, Mar 19, 2013 at 5:42 AM, Anupam Ranjan <
    fantasticanupam@gmail.com> wrote:
    Hi All,

    There is an issue with Namenode on Cloudera Manager. Whenever I start
    the Namenode, It is not starting and giving this error message in log. The
    same issue persist with HBase too.

    Error:

    Supervisor returned FATAL: + '[' -e /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
    + perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g' /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    ++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth 1 -name '*.py'
    + OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
    + '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
    + chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
    + export HADOOP_IDENT_STRING=hdfs
    + HADOOP_IDENT_STRING=hdfs
    + '[' -n '' ']'
    + acquire_kerberos_tgt hdfs.keytab
    + '[' -z hdfs.keytab ']'
    + '[' -n '' ']'
    + '[' validate-writable-empty-dirs = namenode ']'
    + '[' file-operation = namenode ']'
    + '[' bootstrap = namenode ']'
    + '[' failover = namenode ']'
    + '[' transition-to-active = namenode ']'
    + '[' initializeSharedEdits = namenode ']'
    + '[' initialize-znode = namenode ']'
    + '[' format-namenode = namenode ']'
    + '[' monitor-decommission = namenode ']'
    + '[' monitor-upgrade = namenode ']'
    + '[' finalize-upgrade = namenode ']'
    + '[' mkdir = namenode ']'
    + '[' namenode = namenode -o secondarynamenode = namenode -o datanode = namenode ']'
    + HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + exec /usr/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


    Thanks & Regards,
    *Anupam Ranjan*


    --
    Thanks,
    Darren

    --
    Thanks,
    Darren
  • J at Apr 9, 2013 at 7:09 am
    Hi guys,

    I have very similar issue like Anupam's.

    I've been running services, then I installed zookeeper & hbase. zookeeper
    was ok but hbase not.

    I'm using 3 servers, for hbase I installed thrift for all 3, too.

    /var/log/hbase/hbase-cmf-hbase1-MASTER-....log.out shows like below;
    ...
    2013-04-09 15:31:35,518 ERROR
    org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
    java.lang.RuntimeException: Failed construction of Master: class
    org.apache.hadoop.hbase.master.HMaster at
    org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1824)
    at
    org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:152)
    at
    org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at
    org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at
    org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1838)Caused by:
    java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method) at
    sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.java:247)
    at
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.(HBaseServer.java:1533)
    at
    org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.(WritableRpcEngine.java:245)
    at
    org.apache.hadoop.hbase.ipc.WritableRpcEngine.getServer(WritableRpcEngine.java:55)
    at
    org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:433) at
    org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:422)
    at
    org.apache.hadoop.hbase.master.HMaster.(Native Method)
    at
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at
    org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1819)
    ... 5 more
    ...

    Thanks,
    Jun

    2013년 3월 19일 화요일 오후 9시 42분 53초 UTC+9, Anupam Ranjan 님의 말:
    Hi All,

    There is an issue with Namenode on Cloudera Manager. Whenever I start the
    Namenode, It is not starting and giving this error message in log. The same
    issue persist with HBase too.

    Error:

    Supervisor returned FATAL: + '[' -e /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py ']'
    + perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE#g' /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    ++ find /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE -maxdepth 1 -name '*.py'
    + OUTPUT='/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py'
    + '[' '/var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py
    /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py' '!=' '' ']'
    + chmod +x /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/cloudera_manager_agent_fencer.py /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE/topology.py
    + export 'HADOOP_OPTS=-Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Djava.net.preferIPv4Stack=true '
    + export HADOOP_IDENT_STRING=hdfs
    + HADOOP_IDENT_STRING=hdfs
    + '[' -n '' ']'
    + acquire_kerberos_tgt hdfs.keytab
    + '[' -z hdfs.keytab ']'
    + '[' -n '' ']'
    + '[' validate-writable-empty-dirs = namenode ']'
    + '[' file-operation = namenode ']'
    + '[' bootstrap = namenode ']'
    + '[' failover = namenode ']'
    + '[' transition-to-active = namenode ']'
    + '[' initializeSharedEdits = namenode ']'
    + '[' initialize-znode = namenode ']'
    + '[' format-namenode = namenode ']'
    + '[' monitor-decommission = namenode ']'
    + '[' monitor-upgrade = namenode ']'
    + '[' finalize-upgrade = namenode ']'
    + '[' mkdir = namenode ']'
    + '[' namenode = namenode -o secondarynamenode = namenode -o datanode = namenode ']'
    + HADOOP_OPTS='-Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + export 'HADOOP_OPTS=-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + HADOOP_OPTS='-Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true '
    + exec /usr/lib/hadoop-hdfs/bin/hdfs --config /var/run/cloudera-scm-agent/process/78-hdfs-NAMENODE namenode


    Thanks & Regards,
    *Anupam Ranjan*
  • Anupam Ranjan at Apr 9, 2013 at 7:14 am
    I still have the same issue...could not get any solution from Cloudera side.


    On 9 April 2013 12:39, j wrote:

    .log.out shows like belo



    *Anupam Ranjan*
  • Philip Zeyliger at Apr 9, 2013 at 5:11 pm
    Hi folks,

    *Caused by: java.net.BindException: Address already in use*
    *
    *
    That error in the logs more or less uniquely means that the port is already
    being used by some other process. Here is how I tend to check what ports
    are being used:

    $sudo lsof -P -n | grep LISTEN | grep 60030
    java 28283 hbase 312u IPv4 148789875 0t0
    TCP *:60030 (LISTEN)

    You should check the configuration to figure out what ports HBase is trying
    to use, and who they're used by. If it's a different HBase process, then
    probably you installed the "hbase-master" package from CDH and it's
    starting HBase for you when your machine boots. Turn that off with
    "chkconfig" and "service hbase-master stop". If it's some other random
    process, you can either kill it or change the port numbers in the CM
    configuration screens.

    Cheers,

    -- Philip

    On Tue, Apr 9, 2013 at 12:13 AM, Anupam Ranjan wrote:

    I still have the same issue...could not get any solution from Cloudera
    side.


    On 9 April 2013 12:39, j wrote:

    .log.out shows like belo



    *Anupam Ranjan*
  • Hyunjun Jeong at Apr 10, 2013 at 12:29 am
    Hi Harsh, and Philip,

    Thanks for your help. I tried to change the port however I couldn't find
    the configuration of port in the manager (Please see attachment though the
    main language is Korean).
    [image: 본문 이미지 1]
    I can see configuration xml files in /var/run/cloudera-scm-agent/process/,
    however I think this is the place for the currently running processes.

    So, I though if I put the port
    configuration /etc/hbase/conf.cloudera.hbase1/hbase-site.xml or
    /etc/hbase/conf.dist/hbase-site.xml, then the value ccould be propagated to
    there but it didn't.

    Let me know where to configure the master port.

    Thanks,
    Jun




    2013/4/10 Philip Zeyliger <philip@cloudera.com>
    Hi folks,

    *Caused by: java.net.BindException: Address already in use*
    *
    *
    That error in the logs more or less uniquely means that the port is
    already being used by some other process. Here is how I tend to check what
    ports are being used:

    $sudo lsof -P -n | grep LISTEN | grep 60030
    java 28283 hbase 312u IPv4 148789875 0t0
    TCP *:60030 (LISTEN)

    You should check the configuration to figure out what ports HBase is
    trying to use, and who they're used by. If it's a different HBase process,
    then probably you installed the "hbase-master" package from CDH and it's
    starting HBase for you when your machine boots. Turn that off with
    "chkconfig" and "service hbase-master stop". If it's some other random
    process, you can either kill it or change the port numbers in the CM
    configuration screens.

    Cheers,

    -- Philip

    On Tue, Apr 9, 2013 at 12:13 AM, Anupam Ranjan wrote:

    I still have the same issue...could not get any solution from Cloudera
    side.


    On 9 April 2013 12:39, j wrote:

    .log.out shows like belo



    *Anupam Ranjan*
  • Darren Lo at Apr 10, 2013 at 12:56 am
    Hi Jun,

    If you expand the Master role config group on the left, you'll see
    something called Ports and Addresses. The configuration for the HBase
    Master port is in there. You can also search for "port" in the search bar
    on the left, above the green checkmark.

    Thanks,
    Darren

    On Tue, Apr 9, 2013 at 5:29 PM, Hyunjun Jeong wrote:

    Hi Harsh, and Philip,

    Thanks for your help. I tried to change the port however I couldn't find
    the configuration of port in the manager (Please see attachment though the
    main language is Korean).
    [image: 본문 이미지 1]
    I can see configuration xml files in /var/run/cloudera-scm-agent/process/,
    however I think this is the place for the currently running processes.

    So, I though if I put the port
    configuration /etc/hbase/conf.cloudera.hbase1/hbase-site.xml or
    /etc/hbase/conf.dist/hbase-site.xml, then the value ccould be propagated to
    there but it didn't.

    Let me know where to configure the master port.

    Thanks,
    Jun




    2013/4/10 Philip Zeyliger <philip@cloudera.com>
    Hi folks,

    *Caused by: java.net.BindException: Address already in use*
    *
    *
    That error in the logs more or less uniquely means that the port is
    already being used by some other process. Here is how I tend to check what
    ports are being used:

    $sudo lsof -P -n | grep LISTEN | grep 60030
    java 28283 hbase 312u IPv4 148789875 0t0
    TCP *:60030 (LISTEN)

    You should check the configuration to figure out what ports HBase is
    trying to use, and who they're used by. If it's a different HBase process,
    then probably you installed the "hbase-master" package from CDH and it's
    starting HBase for you when your machine boots. Turn that off with
    "chkconfig" and "service hbase-master stop". If it's some other random
    process, you can either kill it or change the port numbers in the CM
    configuration screens.

    Cheers,

    -- Philip


    On Tue, Apr 9, 2013 at 12:13 AM, Anupam Ranjan <fantasticanupam@gmail.com
    wrote:
    I still have the same issue...could not get any solution from Cloudera
    side.


    On 9 April 2013 12:39, j wrote:

    .log.out shows like belo



    *Anupam Ranjan*

    --
    Thanks,
    Darren
  • Hyunjun Jeong at Apr 10, 2013 at 8:04 am
    Thanks Darren, I found it :)


    2013/4/10 Darren Lo <dlo@cloudera.com>
    Hi Jun,

    If you expand the Master role config group on the left, you'll see
    something called Ports and Addresses. The configuration for the HBase
    Master port is in there. You can also search for "port" in the search bar
    on the left, above the green checkmark.

    Thanks,
    Darren

    On Tue, Apr 9, 2013 at 5:29 PM, Hyunjun Jeong wrote:

    Hi Harsh, and Philip,

    Thanks for your help. I tried to change the port however I couldn't find
    the configuration of port in the manager (Please see attachment though the
    main language is Korean).
    [image: 본문 이미지 1]
    I can see configuration xml files
    in /var/run/cloudera-scm-agent/process/, however I think this is the place
    for the currently running processes.

    So, I though if I put the port
    configuration /etc/hbase/conf.cloudera.hbase1/hbase-site.xml or
    /etc/hbase/conf.dist/hbase-site.xml, then the value ccould be propagated to
    there but it didn't.

    Let me know where to configure the master port.

    Thanks,
    Jun




    2013/4/10 Philip Zeyliger <philip@cloudera.com>
    Hi folks,

    *Caused by: java.net.BindException: Address already in use*
    *
    *
    That error in the logs more or less uniquely means that the port is
    already being used by some other process. Here is how I tend to check what
    ports are being used:

    $sudo lsof -P -n | grep LISTEN | grep 60030
    java 28283 hbase 312u IPv4 148789875
    0t0 TCP *:60030 (LISTEN)

    You should check the configuration to figure out what ports HBase is
    trying to use, and who they're used by. If it's a different HBase process,
    then probably you installed the "hbase-master" package from CDH and it's
    starting HBase for you when your machine boots. Turn that off with
    "chkconfig" and "service hbase-master stop". If it's some other random
    process, you can either kill it or change the port numbers in the CM
    configuration screens.

    Cheers,

    -- Philip


    On Tue, Apr 9, 2013 at 12:13 AM, Anupam Ranjan <
    fantasticanupam@gmail.com> wrote:
    I still have the same issue...could not get any solution from Cloudera
    side.


    On 9 April 2013 12:39, j wrote:

    .log.out shows like belo



    *Anupam Ranjan*

    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 19, '13 at 12:42p
activeApr 10, '13 at 8:04a
posts11
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase