I have tried to running CM4 to install everything onto 6 nodes but get
heatbeat failed messages along with other install failure messages.
Please find attached my errors and some more info.
There is no firewalling on our LAN.
all done
closing logging file descriptor
[12/Feb/2013:10:38:31] ENGINE Shutting down due to error in start listener:
"/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
line 235, in start
"/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/wspbus.py",
line 215, in publish
[12/Feb/2013:10:38:31] ENGINE Bus STOPPING
[12/Feb/2013:10:38:31] ENGINE HTTP Server
cherrypy._cpwsgi_server.CPWSGIServer(('cloudera-node2', 9000)) already shut
down
[12/Feb/2013:10:38:31] ENGINE Stopped thread '_TimeoutMonitor'.
[12/Feb/2013:10:38:31] ENGINE Bus STOPPED
[12/Feb/2013:10:38:31] ENGINE Bus EXITING
[12/Feb/2013:10:38:31] ENGINE Bus EXITED
vars
as pid 24672 user root(0) group root(0).
pre-existing directory: /var/run/cloudera-scm-agent
path: /opt/cloudera/parcel-cache
/opt/cloudera/parcel-cache for parcel cache
pre-existing directory: /var/run/cloudera-scm-agent/cgroups
/var/run/cloudera-scm-agent/cgroups/memory
/var/run/cloudera-scm-agent/cgroups/cpu
/var/run/cloudera-scm-agent/cgroups/blkio
capabilities: {'has_memory': True, 'default_memory_limit_in_bytes':
9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}
supervisord event monitor.
Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
timeout of 2.000000
join timeout of 0.100000
tolerance of 60.000000
metrics_url_timeout_seconds of 30.000000
task_metrics_timeout_seconds of 5.000000
max_collection_wait_seconds of 10.000000
tasktracker metric schema from file
/usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
Failed to collect disk stats for device 'mapper/VolGroup-lv_root'
54, in collect_update
'/sys/block/mapper/VolGroup-lv_root/stat'
Failed to add DiskDevice for disk='mapper/VolGroup-lv_root'
metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
processes will inherit the following environment: {'CMF_CONF_DIR':
'/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CMF_AGENT_MGMT_HOME':
'/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CMF_AGENT_CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CMF_AGENT_CDH_HUE_HOME':
'/usr/share/hue', 'SHLVL': '1', 'CMF_AGENT_TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',
'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
'/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN': '/usr/bin/hadoop',
'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CMF_AGENT_CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
'/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
'/usr/lib/hadoop-hdfs', 'CMF_AGENT_CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CMF_AGENT_CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'PWD': '/', 'CMF_AGENT_CDH_HADOOP_HOME':
'/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
processes will add the following their environment environment:
{'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
'/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME':
'/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
'/usr/lib/hive', 'JSVC_HOME': '/usr/libexec/bigtop-utils',
'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME':
'/usr/lib/hbase', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME':
'/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
'/usr/lib/pig', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
pre-existing directory: /var/run/cloudera-scm-agent/process
pre-existing directory: /var/run/cloudera-scm-agent/supervisor
pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
previous supervisor: agent-8767-1360595961.
[12/Feb/2013:10:46:14] ENGINE Bus STARTING
[12/Feb/2013:10:46:14] ENGINE Started monitor thread '_TimeoutMonitor'.
IOError: Port 9000 not free on 'cloudera-node2'
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging ERROR
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging ERROR
Traceback (most recent call last):
File
File
line 235, in start
self.publish('start')
File
File
line 215, in publish
raise exc
ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
ChannelFailures: IOError("Port 9000 not free on 'cloudera-node2'",)
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
cherrypy._cpwsgi_server.CPWSGIServer(('cloudera-node2', 9000)) already shut
down
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013 10:38:31 +0000] 23655 MainThread _cplogging INFO
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO No command line
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Agent starting
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread downloader INFO Downloader
[12/Feb/2013 10:46:12 +0000] 24672 MainThread parcel_cache INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Re-using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
[12/Feb/2013 10:46:12 +0000] 24672 MainThread cgroups INFO Reusing
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Found cgroups
9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True,
'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Setting up
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread filesystem_map INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread agent INFO Using
[12/Feb/2013 10:46:12 +0000] 24672 MainThread metrics INFO Importing
/usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[12/Feb/2013 10:46:12 +0000] 24672 MainThread throttling_logger ERROR
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/disk_device.py", line
File "/usr/lib64/cmf/agent/src/cmf/monitor/host/disk_device.py", line
stat_file = open(self._stat_path)
IOError: [Errno 2] No such file or directory:
IOError: [Errno 2] No such file or directory:
[12/Feb/2013 10:46:12 +0000] 24672 MainThread disk_devices WARNING
[12/Feb/2013 10:46:12 +0000] 24672 MainThread __init__ INFO Importing
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
'/etc/cloudera-scm-agent', 'CMF_AGENT_HIVE_DEFAULT_XML':
'/etc/hive/conf.dist/hive-default.xml', 'CMF_AGENT_MGMT_HOME':
'/usr/share/cmf', 'CMF_AGENT_CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CMF_AGENT_CDH_FLUME_HOME':
'/usr/lib/flume-ng', 'CMF_AGENT_CDH_HBASE_HOME': '/usr/lib/hbase', 'LANG':
'en_US.UTF-8', 'TERM': 'vt100', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CMF_AGENT_CDH_HUE_HOME':
'/usr/share/hue', 'SHLVL': '1', 'CMF_AGENT_TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar',
'CMF_AGENT_CDH_PIG_HOME': '/usr/lib/pig', 'CMF_AGENT_CDH_HIVE_HOME':
'/usr/lib/hive', 'CLOUDERA_MYSQL_CONNECTOR_JAR':
'/usr/share/java/mysql-connector-java.jar', 'CMF_AGENT_CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CMF_AGENT_CDH_HADOOP_BIN': '/usr/bin/hadoop',
'CMF_AGENT_CDH_OOZIE_HOME': '/usr/lib/oozie',
'CMF_AGENT_CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper',
'CMF_AGENT_CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CMF_AGENT_JSVC_HOME':
'/usr/libexec/bigtop-utils', 'CMF_AGENT_CDH_HDFS_HOME':
'/usr/lib/hadoop-hdfs', 'CMF_AGENT_CDH_MR1_HOME':
'/usr/lib/hadoop-0.20-mapreduce', 'CMF_AGENT_CDH_MR2_HOME':
'/usr/lib/hadoop-mapreduce', 'PWD': '/', 'CMF_AGENT_CDH_HADOOP_HOME':
'/usr/lib/hadoop', 'CMF_AGENT_CDH_IMPALA_HOME': '/usr/lib/impala'}
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Supervised
{'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CMF_PACKAGE_DIR':
'/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME':
'/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME':
'/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs',
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE
HDFS_DFS_DIR_NOT_EMPTY', 'CLOUDERA_ORACLE_CONNECTOR_JAR':
'/usr/share/java/oracle-connector-java.jar', 'CDH_HIVE_HOME':
'/usr/lib/hive', 'JSVC_HOME': '/usr/libexec/bigtop-utils',
'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml',
'CLOUDERA_POSTGRESQL_JDBC_JAR':
'/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_HOME':
'/usr/lib/hbase', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME':
'/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/share/hue',
'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar',
'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_PIG_HOME':
'/usr/lib/pig', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs',
'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'TOMCAT_HOME':
'/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Re-using
[12/Feb/2013 10:46:14 +0000] 24672 MainThread agent INFO Connecting to
[12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
[12/Feb/2013 10:46:14 +0000] 24672 MainThread _cplogging INFO
IP Tables are OFF
SELinux is DISABLED
Netstat readout
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign
Address State PID/Program name
tcp 0 0 cloudera-node1:9000
*:* LISTEN 17258/python
tcp 0 0 *:7432
*:* LISTEN 1798/postgres
tcp 0 0 localhost:9001
*:* LISTEN 8945/python2.6
tcp 0 0 *:7180
*:* LISTEN 1826/java
tcp 0 0 *:7182
*:* LISTEN 1826/java
tcp 0 0 *:111
*:* LISTEN 1426/rpcbind
tcp 0 0 *:22
*:* LISTEN
tcp 0 0 *:54102
*:* LISTEN
tcp 0 0 localhost:36186
localhost:9001 ESTABLISHED
tcp 0 0 cloudera-node1:38815
cloudera-node3:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58248 ESTABLISHED
tcp 0 0 cloudera-node1:7180 webvpn.domain.co.uk:16971
ESTABLISHED
tcp 0 0 localhost:58245
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58247
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:39077
cloudera-node1:7182 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58246 ESTABLISHED
tcp 0 0 cloudera-node1:22 webvpn.domain.co.uk:16062
ESTABLISHED
tcp 0 0 cloudera-node1:43521
cloudera-node6:22 ESTABLISHED
tcp 0 0 localhost:58246
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58248
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:49106
cloudera-node5:22 ESTABLISHED
tcp 0 0 cloudera-node1:49705
cloudera-node4:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58244 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58247 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58245 ESTABLISHED
tcp 0 0 localhost:9001
localhost:36186 ESTABLISHED
tcp 0 0 cloudera-node1:7182
cloudera-node1:39077 ESTABLISHED
tcp 0 0 localhost:58244
localhost:7432 ESTABLISHED
tcp 0 0 *:7432
*:* LISTEN
tcp 0 0 *:33514
*:* LISTEN
tcp 0 0 *:111
*:* LISTEN
tcp 0 0 *:22
*:* LISTEN
Address State PID/Program name
tcp 0 0 cloudera-node1:9000
*:* LISTEN 17258/python
tcp 0 0 *:7432
*:* LISTEN 1798/postgres
tcp 0 0 localhost:9001
*:* LISTEN 8945/python2.6
tcp 0 0 *:7180
*:* LISTEN 1826/java
tcp 0 0 *:7182
*:* LISTEN 1826/java
tcp 0 0 *:111
*:* LISTEN 1426/rpcbind
tcp 0 0 *:22
*:* LISTEN
tcp 0 0 *:54102
*:* LISTEN
tcp 0 0 localhost:36186
localhost:9001 ESTABLISHED
tcp 0 0 cloudera-node1:38815
cloudera-node3:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58248 ESTABLISHED
tcp 0 0 cloudera-node1:7180 webvpn.domain.co.uk:16971
ESTABLISHED
tcp 0 0 localhost:58245
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58247
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:39077
cloudera-node1:7182 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58246 ESTABLISHED
tcp 0 0 cloudera-node1:22 webvpn.domain.co.uk:16062
ESTABLISHED
tcp 0 0 cloudera-node1:43521
cloudera-node6:22 ESTABLISHED
tcp 0 0 localhost:58246
localhost:7432 ESTABLISHED
tcp 0 0 localhost:58248
localhost:7432 ESTABLISHED
tcp 0 0 cloudera-node1:49106
cloudera-node5:22 ESTABLISHED
tcp 0 0 cloudera-node1:49705
cloudera-node4:22 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58244 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58247 ESTABLISHED
tcp 0 0 localhost:7432
localhost:58245 ESTABLISHED
tcp 0 0 localhost:9001
localhost:36186 ESTABLISHED
tcp 0 0 cloudera-node1:7182
cloudera-node1:39077 ESTABLISHED
tcp 0 0 localhost:58244
localhost:7432 ESTABLISHED
tcp 0 0 *:7432
*:* LISTEN
tcp 0 0 *:33514
*:* LISTEN
tcp 0 0 *:111
*:* LISTEN
tcp 0 0 *:22
*:* LISTEN
Any thoughts on how to resolve this?